Papers with Speech & Audio in NLP
Learning to Understand Child-directed and Adult-directed Speech (2020.acl-main)
Copied to clipboard
| Challenge: | linguistic properties of child-directed speech differ from adult-directed in many ways . linguistic differences between CDS and ADS are retained, but the acoustic properties are similar. |
| Approach: | They compare the task performance of models trained on adult-directed speech and child-directed language . they propose that CDS is optimized for learnability, but not for comprehension . |
| Outcome: | The proposed model trains on adult-directed speech and child-directed language . the model generalizes better on the training register and on synthesized speech . |
End-to-End Evaluation for Low-Latency Simultaneous Speech Translation (2023.emnlp-demo)
Copied to clipboard
Christian Huber, Tu Anh Dinh, Carlos Mullov, Ngoc-Quan Pham, Thai Binh Nguyen, Fabian Retkowski, Stefan Constantin, Enes Ugan, Danni Liu, Zhaolin Li, Sai Koneru, Jan Niehues, Alexander Waibel
| Challenge: | a framework to evaluate low-latency speech translations is currently only limited to specific aspects and is not able to compare different approaches. |
| Approach: | They propose a framework to perform and evaluate low-latency speech translation in realistic conditions. |
| Outcome: | The proposed framework evaluates various aspects of low-latency speech translation under realistic conditions. |
Emotion Impacts Speech Recognition Performance (N19-3)
Copied to clipboard
| Challenge: | Existing studies show that speech recognition systems depend on multiple factors including lexical content, speaker identity and dialect. |
| Approach: | They propose a method that evaluates the impact of emotion on recognition even when manual transcripts are not available. |
| Outcome: | The proposed method allows to evaluate the impact of emotion on recognition even when manual transcripts are not available. |
PolyNorm: Few-Shot LLM-Based Text Normalization for Text-to-Speech (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Text Normalization (TN) is a key preprocessing step in Text-to-Speech systems. |
| Approach: | They propose a prompt-based approach to TN using Large Language Models (LLMs) they propose scalable experimentation across languages to reduce the reliance on manual rules . |
| Outcome: | The proposed approach reduces the reliance on manual rules and enables broader linguistic applicability with minimal human intervention across eight languages. |
CoSSAT: Code-Switched Speech Annotation Tool (D19-59)
Copied to clipboard
| Challenge: | Code-switching is a phenomenon that occurs in multilingual societies where speakers who are fluent in two or more languages switch between these languages in the same conversation or utterance. |
| Approach: | They propose an interface which helps annotators transcribe code-switched speech faster, more easily and more accurately than a traditional interface. |
| Outcome: | The proposed interface can be used by 10 users to transcribe Hindi-English code-switched speech faster, easier and more accurately than a traditional interface. |
Chameleon: A Language Model Adaptation Toolkit for Automatic Speech Recognition of Conversational Speech (D19-3)
Copied to clipboard
| Challenge: | Language model adaptation (LMA) is a promising solution for conversational speech recognition systems. |
| Approach: | They propose to use language model adaptation techniques to adapt language models to conversational speech recognition. |
| Outcome: | The proposed toolkit compares state-of-the-art language model adaptation techniques in conversational speech recognition tasks. |
Atypical Inputs in Educational Applications (N18-3)
Copied to clipboard
| Challenge: | atypical characteristics of some responses make it difficult for an automated scoring system to assign a valid score . a typical spoken response with a lot of background noise may suffer from frequent errors in automated speech recognition . |
| Approach: | They propose a pipeline that detects and processes non-scorable responses at run-time . they also propose linguistic filtering models for spoken responses in language tests . |
| Outcome: | The proposed pipeline detects and processes non-scorable responses at run-time and evaluates them for spoken responses in language proficiency assessment. |
Massively Multilingual Adversarial Speech Recognition (N19-1)
Copied to clipboard
| Challenge: | Prior work in multilingual and cross-lingual speech recognition has been limited to a subset of the world's most-spoken languages. |
| Approach: | They propose to use phonemes and phonemes as pretraining objectives to encourage language-independent representations. |
| Outcome: | The proposed model is able to learn language-independent representations of speech using multilingual training. |
Personalized Dense Retrieval on Global Index for Voice-enabled Conversational Systems (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Constrained retrieval is limited to entities in recent user history, which offers low coverage of future requests. |
| Approach: | They propose a personalized entity retrieval system that is robust to phonetic noise and ambiguity but is not limited to a customized index. |
| Outcome: | The proposed system corrects multiple error modes and shows 91% improvement over baseline on the entity retrieval task. |
Proteno: Text Normalization with Limited Data for Fast Deployment in Text to Speech Systems (2021.naacl-industry)
Copied to clipboard
| Challenge: | Developing Text Normalization systems for Text-to-Speech (TTS) on new languages is hard. |
| Approach: | They propose a novel architecture to facilitate Text Normalization systems for TTS on new languages . they use a granular tokenization mechanism that enables the system to learn majority of classes . |
| Outcome: | The proposed architecture performs comparable with the state-of-the-art systems on English . the proposed system learns most classes from training data and precodes them for other classes . |
VoiceBench: Benchmarking LLM-Based Voice Assistants (2026.tacl-1)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have enabled real-time speech interactions through LLMs. |
| Approach: | They propose a benchmark specifically designed to assess LLM-based voice assistants. |
| Outcome: | The proposed benchmark measures the performance of LLM-based voice assistants across eight tasks. |
RoDia: A New Dataset for Romanian Dialect Identification from Speech (2024.findings-naacl)
Copied to clipboard
| Challenge: | a dataset for Romanian dialect identification from speech is released . the dataset includes speech samples from five distinct regions of Romania . |
| Approach: | They propose a dataset for Romanian dialect identification from speech . they propose competitive models to be used as baselines for future research . |
| Outcome: | The first dataset for Romanian dialect identification from speech is released . the top scoring model achieves 59.83% and 62.08%, respectively . |
Follow-on Question Suggestion via Voice Hints for Voice Assistants (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Query suggestion is a standard feature of screen-based search experiences, but it is not trivial to implement in voice-based settings. |
| Approach: | They propose a task of suggesting questions with compact voice hints to allow users to ask follow-up questions. |
| Outcome: | The proposed approach is based on a dataset of 6681 input questions and human written hints and is highly linguistically motivated. |
Neural Text Normalization with Subword Units (N19-2)
Copied to clipboard
| Challenge: | Text normalization (TN) is an important step in conversational systems. |
| Approach: | They frame text normalization as a machine translation task and tackle it with sequence-to-sequence models. |
| Outcome: | The proposed model normalizes written text to its spoken form to facilitate speech recognition and text-to-speech synthesis. |
Audio De-identification - a New Entity Recognition Task (N19-2)
Copied to clipboard
Ido Cohn, Itay Laish, Genady Beryozkin, Gang Li, Izhak Shafran, Idan Szpektor, Tzvika Hartman, Avinatan Hassidim, Yossi Matias
| Challenge: | Named Entity Recognition (NER) is an important step in de-identification (de-ID) of medical records, many of which are recorded conversations between a patient and a doctor. |
| Approach: | They propose to use Named Entity Recognition (NER) to detect audio spans with entity mentions in medical records and then use it to evaluate the results. |
| Outcome: | The proposed pipeline is based on a large labeled segment of the Switchboard and Fisher audio datasets and compares it with a benchmark. |
Speech acts and Communicative Intentions for Urgency Detection (2022.starsem-1)
Copied to clipboard
| Challenge: | Existing approaches to detect speech acts (SA) in synchronous and asynchronous dialogues have been proposed to capture communicative intentions on the part of the speaker. |
| Approach: | They propose to annotate tweets with urgency and SA and develop deep learning architectures to inject it into urgency detection. |
| Outcome: | The proposed dataset annotated for urgency and SA improves information type detection in an out-of-type configuration where models are evaluated in unseen event types during training. |
Discourse on ASR Measurement: Introducing the ARPOCA Assessment Tool (2022.acl-srw)
Copied to clipboard
| Challenge: | Automated speech recognition (ASR) models are based on a corpus of audio recordings, but are often small or nonexistent for less common languages and dialects. |
| Approach: | This research proposal will develop a semi-automatic acoustic features extraction system that integrates phonetic transcripts with pronunciation dictionaries. |
| Outcome: | The proposed system will be used to improve language recognition and model feedback in less common languages and dialects. |
CarMem: Enhancing Long-Term Memory in LLM Voice Assistants through Category-Bounding (2025.coling-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are stateless and present all relevant memories during each interaction, resulting in repetitive user requests and disengagement. |
| Approach: | They propose a long-term memory system for voice assistants structured around predefined categories that leverages Large Language Models to extract, store, and retrieve preferences within these categories. |
| Outcome: | The proposed system achieves an F1-score of .78 to .95 in preference extraction, depending on category granularity, and is suitable for industrial applications. |
On the Locality of Attention in Direct Speech Translation (2022.acl-srw)
Copied to clipboard
| Challenge: | Recent advances in NLP have created problems with the complexity of the self-attention layer. |
| Approach: | They propose to substitute standard self-attention with a local efficient one to avoid the computation of attention weights. |
| Outcome: | The proposed model matches the baseline performance and improves efficiency by skipping the computation of weights that standard attention discards. |
Wav2Gloss: Generating Interlinear Glossed Text from Speech (2024.acl-long)
Copied to clipboard
Taiqi He, Kwanghee Choi, Lindia Tjuatja, Nathaniel Robinson, Jiatong Shi, Shinji Watanabe, Graham Neubig, David Mortensen, Lori Levin
| Challenge: | Interlinear Glossed Text (IGT) is a form of linguistic annotation that can support documentation and resource creation for endangered languages. |
| Approach: | They propose a task in which these four annotation components are extracted automatically from speech and introduce a dataset to lay the groundwork for future research on IGT generation from speech. |
| Outcome: | The proposed dataset provides the first dataset to lay the groundwork for future research on IGT generation from speech, including end-to-end versus cascaded, monolingual versus multilingual, and single-task versus multiple-task approaches. |
Automatic Named Entity Obfuscation in Speech (2023.findings-acl)
Copied to clipboard
| Challenge: | Identifying, replacing and inserting replacement named entities synthesized using voice cloning into original audio reduces the likelihood of deanonymization. |
| Approach: | They propose to use a named entity recognition system built directly on speech to replace a masked language model and synthesize the replacement named entities using voice cloning. |
| Outcome: | The proposed system is built on the English speech NER dataset and tested on a sample of the LibriSpeech corpus. |
The taste of IPA: Towards open-vocabulary keyword spotting and forced alignment in any language (2024.naacl-long)
Copied to clipboard
| Challenge: | a recent study shows that multilingual speech processing systems can generalize to unseen languages without adaptation. |
| Approach: | They propose a phoneme-based phoneme embedding model that can be generalized to unseen languages by using a neural forced aligner. |
| Outcome: | The proposed model can generalize to unseen languages without adaptation. |
DuIVRS-2: An LLM-based Interactive Voice Response System for Large-scale POI Attribute Acquisition (2026.acl-industry)
Copied to clipboard
| Challenge: | Accurate Point of Interest (POI) attribute acquisition is essential for location-based services, yet traditional IVR systems suffer from error accumulation and high maintenance overhead. |
| Approach: | They propose a large language model-based framework for large-scale POI attribute acquisition at Baidu Maps. |
| Outcome: | The proposed framework outperforms existing IVR systems in 83.9% task success rate while maintaining a low reaction time of 130ms. |
Centering the Speech Community (2024.eacl-long)
Copied to clipboard
| Challenge: | In remote speech communities, people interact with the outside world using a variety of an institutional language. |
| Approach: | They propose to use local languages to support their collaboration in a remote community in the far north of australia to explore the functional differences between oral and institutional languages. |
| Outcome: | The proposed language technologies are better aligned with local interests and aspirations than the first author's western framing of language as data for exploitation by machines. |
SpeechWeave: Diverse Multilingual Synthetic Text & Audio Data Generation Pipeline for Training Text to Speech Models (2025.acl-industry)
Copied to clipboard
| Challenge: | Text-to-Speech (TTS) training requires extensive and diverse text and speech data. |
| Approach: | They propose a synthetic speech data generation pipeline that generates multilingual, domain-specific datasets for TTS training. |
| Outcome: | The proposed pipeline generates data that is 10–48% more diverse than baseline across various linguistic and phonetic metrics, along with speaker-standardized speech audio while generating approximately 97% correctly normalized text. |
On the Use of External Data for Spoken Named Entity Recognition (2022.naacl-main)
Copied to clipboard
| Challenge: | Named entity recognition (NER) tasks require large labeled datasets to perform . compared to prior work, relative improvements in F1 of up to 16% are found . |
| Approach: | They propose to use self-training, knowledge distillation, and transfer learning to learn SLU models . they compare pipeline and pipeline approaches to find out how to use external data . |
| Outcome: | The proposed models improve performance beyond pre-trained models in resource-constrained settings . the best baseline model is a pipeline approach, while the best performance is achieved by an E2E model. |
CBFC: a parallel L2 speech corpus for Korean and French learners (L18-1)
Copied to clipboard
| Challenge: | Using corpora for second language acquisition has become more and more common . corporata are used to study morpho-syntactic phenomena in English as a foreign language . |
| Approach: | They propose to use a bilingual corpus of French learners of Korean and Korean learners of French to provide a translated and annotated corpus to the scientific community. |
| Outcome: | The proposed corpus can be used for a wide array of purposes in the field of theoretical but also applied linguistics. |
Deep Learning against COVID-19: Respiratory Insufficiency Detection in Brazilian Portuguese Speech (2021.findings-acl)
Copied to clipboard
Edresson Casanova, Lucas Gris, Augusto Camargo, Daniel da Silva, Murilo Gazzola, Ester Sabino, Anna Levin, Arnaldo Candido Jr, Sandra Aluisio, Marcelo Finger
| Challenge: | Respiratory insufficiency is a symptom that requires hospitalization . a dataset was created to analyze COVID-19 patients and a control group . |
| Approach: | They used a dataset to build a Convolution Neural Network to detect respiratory insufficiency using MFCC representations. |
| Outcome: | The proposed method achieves 91.66% accuracy under real-life environmental conditions. |
CTC-based Compression for Direct Speech Translation (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing studies have shown that a dynamic phone-informed compression of the input audio is beneficial for speech translation (ST). |
| Approach: | They propose a method which performs a phone-informed compression of the input audio in direct ST models by exploiting the Connectionist Temporal Classification (CTC) they demonstrate that their method brings a 1.3-1.5 BLEU improvement over a strong baseline on two language pairs (English-Italian and English-German) |
| Outcome: | The proposed method brings a 1.3-1.5 BLEU improvement over a strong baseline on two language pairs (English-Italian and English-German) it reduces memory footprint by more than 10%, and is faster than previous approaches. |
DP-Parse: Finding Word Boundaries from Raw Speech with an Instance Lexicon (2022.tacl-1)
Copied to clipboard
Robin Algayres, Tristan Ricoul, Julien Karadayi, Hugo Laurençon, Salah Zaiem, Abdelrahman Mohamed, Benoît Sagot, Emmanuel Dupoux
| Challenge: | Existing nonparametric models for text segmentation use a Dirichlet process to jointly segment sentences and build a lexicon of word types. |
| Approach: | They propose a Bayesian nonparametric model that uses a Dirichlet process to jointly segment sentences and build a lexicon of word types. |
| Outcome: | The proposed model improves on the Zero Resource Speech Benchmark 2017 and can learn semantic and syntactic representations as assessed by a new spoken word embedding benchmark. |
STEER: Semantic Turn Extension-Expansion Recognition for Voice Assistants (2023.emnlp-industry)
Copied to clipboard
Leon Zhang, Jiarui Lu, Joel Ruben Antony Moniz, Aditya Kulkarni, Dhivya Piraviperumal, Tien Dung Tran, Nick Tzou, Hong Yu
| Challenge: | Existing training datasets for steering use cases are limited due to the cold-start problem. |
| Approach: | They propose a steering detection model that predicts whether a follow-up turn is a user’s attempt to steer the previous command. |
| Outcome: | The proposed model outperforms existing models on human-graded evaluation sets and shows that it can identify steering intent with over 95% accuracy. |
How Users React to Proactive Voice Assistant Behavior While Driving (2020.lrec-1)
Copied to clipboard
| Challenge: | Nowadays Personal Assistants (PAs) are available in multiple environments and become increasingly popular to use via voice. |
| Approach: | They conducted a usability study in which 42 participants perceive proactive voice output in a Wizard-of-Oz study in . traffic density was varied during a highway drive and it included six in-car-specific use cases. |
| Outcome: | The proposed suggestions should not be obtrusive nor increase drivers’ cognitive load, while enhancing user experience. |
ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modeling (2026.acl-long)
Copied to clipboard
| Challenge: | Recent efforts on text-to-audio generation are exploring fine-grained controllability . however, their performance at scale is limited due to data scarcity . |
| Approach: | They propose a multi-task learning problem for high-controllability text-to-audio generation . they propose scalable diffusion transformers that augment condition information in sequence . |
| Outcome: | The proposed method outperforms existing methods on objective and subjective evaluations. |
Thesis Proposal: Self-Adaptive and Epistemic Uncertainty-Guided ASR of Dense Intra-Sentential Code-Switched Speech for African Low-Resource Languages (2026.acl-srw)
Copied to clipboard
| Challenge: | Existing multilingual and pretrained ASR systems improve general recognition accuracy but are weak at switch regions and are sensitive to language imbalance during adaptation. |
| Approach: | They propose a self-adaptive and epistemic uncertainty-guided framework for African low-resource code-switched ASR using Hausa–English and Hausa-Yorùbá as case studies. |
| Outcome: | The proposed framework is based on Hausa–English and Hausa-Yorùbá as case studies. |
Towards Speech Dialogue Translation Mediating Speakers of Different Languages (2023.findings-acl)
Copied to clipboard
| Challenge: | a new task is proposed to mediate speakers of different languages using speech dialogue translation . we consider context as an important aspect that needs to be addressed in this task . speech translation (ST) has also recently shown success in monologue translation - but no study has focused on ST of dialogues . |
| Approach: | They propose a task to mediate speakers of different languages using speech dialogue translation . they construct a speechBSD dataset and conduct baseline experiments . |
| Outcome: | The proposed task mediates speakers of different languages using speech dialogue translation dataset . it shows that bilingual context performs better in our settings . |
“Alexa in the wild” – Collecting Unconstrained Conversations with a Modern Voice Assistant in a Public Environment (2020.lrec-1)
Copied to clipboard
| Challenge: | Currently, many studies on human-machine interactions focus on private usage, short pre-defined tasks or specific domains. |
| Approach: | They propose to collect 40 hours of device directed utterances during a science exhibition in germany and extract transcripts of both visitors requests and Alexa answers. |
| Outcome: | The proposed dataset provides an unconstrained, unscripted public interaction with a voice assistant during a science exhibition in germany. |
Listening Comprehension over Argumentative Content (D18-1)
Copied to clipboard
Shachar Mirkin, Guy Moshkowich, Matan Orbach, Lili Kotlerman, Yoav Kantor, Tamar Lavee, Michal Jacovi, Yonatan Bilu, Ranit Aharonov, Noam Slonim
| Challenge: | In argumentation domain, people are exposed directly to audio (or the video), without access to a written version. |
| Approach: | They present a task for machine listening comprehension in the argumentation domain and a dataset in English. |
| Outcome: | The proposed task is based on 200 speeches arguing for or against 50 controversial topics and uses baseline methods to address it. |
Prosodic segmentation for parsing spoken dialogue (2021.acl-long)
Copied to clipboard
| Challenge: | Existing parsers struggle to parse spoken dialogue because of disfluencies and unmarked boundaries between sentence-like units (SUs). |
| Approach: | They hypothesize that prosody affects a parser that receives an entire dialogue turn as input, instead of gold standard pre-segmented SUs. |
| Outcome: | The proposed model performs better than the SU-based model on the English Switchboard corpus despite performing two tasks rather than one, and pitch and intensity features are the most important for this corpus. |
Hierarchical Policy Optimization for Simultaneous Translation of Unbounded Speech (2026.acl-long)
Copied to clipboard
| Challenge: | Existing synthesis methods cannot guarantee data quality. |
| Approach: | They propose a hierarchical reward that balances translation quality and latency objectives by combining supervised fine-tuning data with supervised inputs. |
| Outcome: | The proposed model can reuse key-value caches across both modalities and eliminate redundant feature recomputation. |
Auto Review: Second Stage Error Detection for Highly Accurate Information Extraction from Phone Conversations (2025.acl-industry)
Copied to clipboard
| Challenge: | Automating benefit verification phone calls saves time and improves patient care. |
| Approach: | They propose a second-stage postprocessing pipeline that reduces manual effort while maintaining a high bar for accuracy. |
| Outcome: | The proposed system significantly reduces manual effort while maintaining a high bar for accuracy while reducing noise and jargon. |
A Human Subject Study of Named Entity Recognition in Conversational Music Recommendation Queries (2023.eacl-main)
Copied to clipboard
| Challenge: | NER is a complex task that requires a high degree of precision and a higher level of recall. |
| Approach: | They evaluated the human NER linguistic behaviour on a noisy corpus of conversational music recommendation queries with many irregular and novel named entities. |
| Outcome: | The results show that human NER was hard to perform under a strict evaluation schema and that the model had higher recall because of entity exposure. |
Generative Pre-trained Speech Language Model with Efficient Hierarchical Transformer (2024.acl-long)
Copied to clipboard
| Challenge: | Experimental results indicate that GPST significantly outperforms the existing speech language models in terms of word error rate, speech quality, and speaker similarity. |
| Approach: | They propose a hierarchical transformer that quantizes audio waveforms into two distinct types of discrete speech representations and integrates them within a transformer architecture. |
| Outcome: | The proposed model outperforms existing speech language models in word error rate, speech quality, and speaker similarity. |
Why Aren’t We NER Yet? Artifacts of ASR Errors in Named Entity Recognition in Spontaneous Speech Transcripts (2023.acl-long)
Copied to clipboard
Piotr Szymański, Lukasz Augustyniak, Mikolaj Morzy, Adrian Szymczak, Krzysztof Surdyk, Piotr Żelasko
| Challenge: | despite advances in language models, the transcript of spontaneous human-human conversations remains an insurmountable challenge for most models. |
| Approach: | They examine the relationship between ASR and NER errors which limit NER models' ability to recover entity mentions from spontaneous speech transcripts. |
| Outcome: | The proposed model fails even if no word errors are introduced by the ASR . the proposed model's performance deteriorates when applied to the ASL outputs . |
Detecting Extraneous Content in Podcasts (2021.eacl-main)
Copied to clipboard
| Challenge: | Podcast episodes often contain extraneous material interleaved within the audio and the written descriptions . authors present classifiers that leverage both textual and listening patterns to detect such content . |
| Approach: | They propose a classifier that leverages both textual and listening patterns to detect extraneous material in podcast descriptions and audio transcripts. |
| Outcome: | The proposed classifiers improve ROUGE scores and reduce extraneous content in podcast summarization tasks. |
Learning Robust and Multilingual Speech Representations (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Unsupervised speech representation learning has shown success at finding representations that correlate with phonetic structures and improve downstream speech recognition performance. |
| Approach: | They evaluate unsupervised speech representation learning representations by looking at their robustness to domain shifts and their ability to improve recognition performance in many languages. |
| Outcome: | The proposed representations improve the recognition performance in 25 phonetically diverse languages and are robust to domain shifts. |
A Speech Recognizer for Frisian/Dutch Council Meetings (2022.lrec-1)
Copied to clipboard
Martijn Bentum, Louis ten Bosch, Henk van den Heuvel, Simone Wills, Domenique van der Niet, Jelske Dijkstra, Hans Van de Velde
| Challenge: | During council meetings both Frisian and Dutch are spoken, and code switching between both languages shows up frequently. |
| Approach: | They develop a bilingual Frisian/Dutch speech recognizer for council meetings in Fryslân (the Netherlands) based on an existing Frisian and Dutch speech recognized by FAME!, which was trained and tested on radio broadcasts. |
| Outcome: | The new recognizer is based on an existing speech recognizer for Frisian and Dutch named FAME!, which was trained and tested on radio broadcasts. |
Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusion (P19-1)
Copied to clipboard
| Challenge: | Existing speech recognition systems are built at individual, isolated utterance level to make building systems computationally feasible. |
| Approach: | They propose to use text-based external word and/or sentence embeddings to integrate conversational context information into a single neural network model. |
| Outcome: | The proposed model outperforms standard end-to-end speech recognition models on the Switchboard conversational speech corpus and improves word error rate with better conversational-context representation. |
Chinese Spoken Named Entity Recognition in Real-world Scenarios: Dataset and Approaches (2024.findings-acl)
Copied to clipboard
| Challenge: | Current Chinese Spoken NER datasets are laboratory-controlled and are limited in topics. |
| Approach: | They propose to use Chinese Spoken NER datasets to extract entities from speech to help voice assistants better grasp the intent behind user's questions and instructions. |
| Outcome: | The proposed methods improve on self-training-asr and mapping then distilling, and even compared with GPT4.0, they achieve better results. |
SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training (2025.findings-acl)
Copied to clipboard
Wenxi Chen, Ziyang Ma, Ruiqi Yan, Yuzhe Liang, Xiquan Li, Ruiyang Xu, Zhikang Niu, Yanqiao Zhu, Yifan Yang, Zhanxun Liu, Kai Yu, Yuxuan Hu, Jinyu Li, Yan Lu, Shujie Liu, Xie Chen
| Challenge: | a new spoken dialogue system with single-stage training is demonstrating its low latency and high quality . SLAM-Omni achieves zero-shot timbre control by modeling spoken language with semantic tokens . |
| Approach: | They propose a timbre-controllable, end-to-end voice interaction system with single-stage training. |
| Outcome: | The proposed system outperforms previous models on 4 GPUs with limited data. |
From Emotion to Expression: Theoretical Foundations and Resources for Fear Speech (2026.eacl-long)
Copied to clipboard
| Challenge: | a new study of fear speech is under-resourced and fragmented. authors review existing definitions and propose a taxonomy that consolidates different dimensions of fear. |
| Approach: | They propose a taxonomy that consolidates different dimensions of fear for studying fear speech. |
| Outcome: | The proposed taxonomy consolidates different dimensions of fear for studying fear speech. |
Speechformer: Reducing Information Loss in Direct Speech Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to speech-to-text translation (ST) use a pipeline of two sub-components - an automatic speech recognition (ASR) and a machine translation (MT) model. |
| Approach: | They propose an architecture that avoids initial lossy compression and aggregates information only at a higher level according to more informed linguistic criteria. |
| Outcome: | The proposed architecture achieves gains of up to 0.8 BLEU on the standard MuST-C corpus and up to 4.0 BLUE in a low resource scenario. |
Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessment (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent phoneme classifiers treat allophonic variation as a single phoneme . atypical pronunciation assessment requires distinguishing between a typical and asymmetric pronunciations . |
| Approach: | They propose a new approach that leverages Gaussian mixture models to model phoneme distributions with multiple subclusters. |
| Outcome: | The proposed approach achieves state-of-the-art across dysarthric and non-native speech datasets. |
Adaptation of Hierarchical Structured Models for Speech Act Recognition in Asynchronous Conversation (N19-1)
Copied to clipboard
| Challenge: | asynchronous domains lack large labeled datasets to train an effective speech act recognition model. |
| Approach: | They propose methods to leverage abundant unlabeled conversational data and available labeled data from synchronous domains to train an effective SAR model. |
| Outcome: | The proposed method outperforms existing methods when trained on in-domain data only. |
Read to Hear: A Zero-Shot Pronunciation Assessment Using Textual Descriptions and LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Automatic pronunciation assessment is typically performed by acoustic models trained on audio-score pairs. |
| Approach: | They propose a zero-shot, textual description-based Pronunciation Assessment approach that utilizes human-readable representations of speech signals fed into an LLM to assess pronunciation accuracy and fluency. |
| Outcome: | The proposed approach is cost-efficient and competitive in performance . it significantly improves the performance of conventional audio-score-trained models on out-of-domain data . |
RedApt: An Adaptor for wav2vec 2 EncodingFaster and Smaller Speech Translation without Quality Compromise (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained speech Transformers in speech translation systems have facilitated state-of-the-art (SotA) results, but their computational cost is high. |
| Approach: | They propose a Reducer Adaptor block that could be seamlessly integrated within any Transformer-based speech encoding architecture. |
| Outcome: | The proposed Reducer Adaptor block outperforms the existing SotA architecture by an average of 0.68 BLEU score on 8 language pairs from Must-C. |
ResoDiff-44k: High-Fidelity Cross-Lingual Speech and Singing Synthesis via Discrete Diffusion (2026.acl-industry)
Copied to clipboard
| Challenge: | generative speech models have a fidelity ceiling that is capped at lower sampling rates . current models rely on intermediate mel-spectrograms, which discard phase and high-frequency information . a new framework that synthesizes industrial-grade 44.1kHz audio is proposed . |
| Approach: | They propose a production-grade generative foundation model for 44.1kHz audio synthesis . they pre-train ResoDiff-44k on a massive 150K -hour multilingual dataset . |
| Outcome: | The proposed model achieves 4.6 mean opinion score in 44.1kHz singing synthesis compared to baselines . it also reduces character error rate on regional mixed-language and singing prompts compared with baselines. |
Hard to Be Heard: Phoneme-Level ASR Analysis of Phonologically Complex, Low-Resource Endangered Languages (2026.findings-acl)
Copied to clipboard
| Challenge: | a phoneme-level analysis of automatic speech recognition (ASR) is performed on two low-resource, typologically complex East Caucasian languages. |
| Approach: | They propose a phoneme-level analysis of automatic speech recognition for two East Caucasian languages, Archi and Rutul. |
| Outcome: | The proposed model improves on existing models and improves in low-resource settings. |
STT4SG-350: A Speech Corpus for All Swiss German Dialect Regions (2023.acl-short)
Copied to clipboard
Michel Plüss, Jan Deriu, Yanick Schraner, Claudio Paonessa, Julia Hartmann, Larissa Schmidt, Christian Scheller, Manuela Hürlimann, Tanja Samardžić, Manfred Vogel, Mark Cieliebak
| Challenge: | We present a corpus of Swiss German speech annotated with Standard German text at the sentence level. |
| Approach: | They present a corpus of Swiss German speech annotated with Standard German sentences . they use a web app to show the speakers standard German sentences and record them . |
| Outcome: | The corpus contains 343 hours of speech from all Swiss German dialect regions . it is the largest public speech corpus for Swiss German to date . |
Establishing degrees of closeness between audio recordings along different dimensions using large-scale cross-lingual models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to analyze speech representations using pretraining data are difficult to achieve for endangered languages. |
| Approach: | They propose an unsupervised method to examine the level of abstraction in vector representations of speech from a pretrained model to determine their level of abstractness. |
| Outcome: | The proposed method is fully unsupervised and could be used in comparative studies on under-documented languages. |
Data Augmentation for Voice-Assistant NLU using BERT-based Interchangeable Rephrase (2021.eacl-main)
Copied to clipboard
| Challenge: | a data augmentation technique is used to boost performance on spoken language understanding tasks. |
| Approach: | They propose a data augmentation technique based on byte pair encoding and a BERT-like self-attention model to boost performance on spoken language understanding tasks. |
| Outcome: | The proposed method performs well on domain and intent classification tasks for a voice assistant and in a user-study focused on utterance naturalness and semantic similarity. |
Efficient CTC Regularization via Coarse Labels for End-to-End Speech Translation (2023.eacl-main)
Copied to clipboard
| Challenge: | Developing techniques to support end-to-end speech translation is non-trivial because of the speech-text modality gap. |
| Approach: | They propose a coarse labeling approach that merges vocabulary labels via simple heuristic rules . they propose to use 256-bit truncation, division or modulo operations to regularize the encoder . |
| Outcome: | The proposed method can increase training efficiency while delivering better performance. |
RiTTA: Modeling Event Relations in Text-to-Audio Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing text-to-audio (TTA) generation methods have not explored audio event relation modeling, nor proposed any new framework to enhance this capability. |
| Approach: | They propose a comprehensive relation corpus covering all potential relations in real-world scenarios and a new audio event corpus encompassing commonly heard audios. |
| Outcome: | The proposed framework improves existing models’ relation modeling capability with negligible extra parameters. |
Aspect Flow Representation and Audio Inspired Analysis for Texts (2020.lrec-1)
Copied to clipboard
| Challenge: | Observing how a text exploits a particular aspect can lead to significant information loss, especially for large texts. |
| Approach: | They propose a method of representing and analysing texts that consider how an aspect behaves throughout the text by aspect flows. |
| Outcome: | The proposed method surpasses summarised features in classification tasks and reveals deeper knowledge about the represented texts. |
SDiaReward: Modeling and Benchmarking Spoken Dialogue Rewards with Modality and Colloquialness (2026.acl-long)
Copied to clipboard
Jingyu Lu, Yuhan Wang, Fan Zhuo, Xize Cheng, Changhao Pan, Xueyi Pu, Yifu Chen, Chenyuhao Wen, Tianle Liang, Zhou Zhao
| Challenge: | SDiaReward is an end-to-end spoken dialogue system that integrates paralinguistic nuances and spontaneous nature of human conversation. |
| Approach: | They propose an end-to-end multi-turn reward model trained on SDiaReward-Dataset . it is a collection of episode-level preference pairs targeting modality and colloquiality gaps . |
| Outcome: | The proposed model outperforms general-purpose audio LLMs in episode-level evaluation. |
Annotation of Emotion Carriers in Personal Narratives (2020.lrec-1)
Copied to clipboard
| Challenge: | PNs are recollections of facts, events, and thoughts that are best explained by speech or text segments . spoken PN is difficult because it is unstructured and involving multiple sub-events and characters as well as thoughts and associated emotions perceived by the narrator. |
| Approach: | They propose and evaluate an annotation model for identifying emotion carriers in spoken personal narratives from the Ulm State-of-Mind in Speech corpus. |
| Outcome: | The proposed model could be used to extract emotion carriers from spoken personal narratives, which are often unstructured and often unorganized . |
AlloSat: A New Call Center French Corpus for Satisfaction and Frustration Analysis (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing systems retrieve emotional information from textual transcriptions or from audio signal. |
| Approach: | They propose to use a call center corpus that is continuously annotated in frustration and satisfaction to model the continuous aspect of semantic and paralinguistic information at the conversation level. |
| Outcome: | The proposed system can model the paralinguistic aspect of semantic and paralinguistic information at the conversation level. |
A Swedish Cookie-Theft Corpus (L18-1)
Copied to clipboard
| Challenge: | Language disturbances can be a diagnostic marker for neurodegenerative diseases, such as Alzheimer's disease, at earlier stages. |
| Approach: | They develop a corpus of audio recordings of the Cookie-theft, a standardized test that has been used in studies in the past. |
| Outcome: | The proposed corpus is based on audio recordings of the Cookie-theft . it provides a rich resource for future research and experimentation in many areas . |
Improving Chinese Pop Song and Hokkien Gezi Opera Singing Voice Synthesis by Enhancing Local Modeling (2023.emnlp-main)
Copied to clipboard
| Challenge: | Singing Voice Synthesis (SVS) synthesizes pleasing vocals based on music scores and lyrics . current acoustic models ignore the significance of local modeling within the sequence and the hard-to-synthesize parts in the predicted mel-spectrogram . |
| Approach: | They propose a method to enhance local modeling in the acoustic model by focusing on phoneme tokens located before and after the phoneme. |
| Outcome: | The proposed method improves local modeling in the acoustic model by focusing on the hard-to-synthesize parts of the predicted mel-spectrogram. |
Investigating the effect of auxiliary objectives for the automated grading of learner English speech transcriptions (2020.acl-main)
Copied to clipboard
| Challenge: | a growing demand for the ability to communicate in English means automated tutoring and assessment systems are becoming more popular. |
| Approach: | They propose to use automatic speech recognition transcripts to grade spontaneous speech based on textual features. |
| Outcome: | The proposed system improves on a transformer encoder with native language identification as an auxiliary task. |
Identification of Primary and Collateral Tracks in Stuttered Speech (2020.lrec-1)
Copied to clipboard
| Challenge: | Disfluency detection is a challenging task because of its different metrics depending on whether the input features are text or speech. |
| Approach: | They propose a framework for disfluency detection inspired by the clinical and the natural language processing perspective together with the theory of performance from (Clark, 1998) . they present a forced-aligned disfluence dataset and propose new audio features inspired by word-based span features. |
| Outcome: | The proposed framework outperforms baselines for speech-based predictions on a forced-aligned disfluency dataset from semi-directed interviews. |
An Automatic Tool For Language Evaluation (2020.lrec-1)
Copied to clipboard
| Challenge: | standardized tests are used to assess and screen developmental language impairments but require manual laborious transcription, annotation and calculation. |
| Approach: | They propose to use the correct sentence and the sentence produced by patients to evaluate the level of verbal production and return a score. |
| Outcome: | The proposed system evaluates the level of the verbal production and returns a score. |
Text Normalization Infrastructure that Scales to Hundreds of Language Varieties (L18-1)
Copied to clipboard
| Challenge: | a multi-language text normalization infrastructure is used to train language models for keyboards and speech recognition systems. |
| Approach: | They describe a multi-language text normalization infrastructure that prepares textual data to train language models used in Google's keyboards and speech recognition systems. |
| Outcome: | The proposed system can normalize training data across hundreds of languages . it can detect errors in training data and detect corruption issues . |
Speech Resources in the Tamasheq Language (2022.lrec-1)
Copied to clipboard
Marcely Zanon Boito, Fethi Bougares, Florentin Barbier, Souhir Gahbiche, Loïc Barrault, Mickael Rouvier, Yannick Estève
| Challenge: | In this paper, we present two datasets for Tamasheq, a developing language mainly spoken in Mali and Niger . we share unlabeled audio data in five languages: french, Fulfulde, Hausa, Tamaheq and Zarma . |
| Approach: | They present two datasets for Tamasheq, a developing language mainly spoken in Mali and Niger. |
| Outcome: | The proposed datasets are used in the IWSLT 2022 low-resource speech translation track . they consist of radio recordings from daily broadcast news in Niger and Mali . |
The French-Algerian Code-Switching Triggered audio corpus (FACST) (L18-1)
Copied to clipboard
| Challenge: | The French Algerian Code-Switching Triggered corpus is a corpus of spontaneous CS utterances . it is used to support linguistic and phonetic studies in phonetics and prosody . |
| Approach: | They propose to use a triggering protocol to elicit CS in natural conversations . they propose to do data segmentation and annotation in each language . |
| Outcome: | The proposed corpus is based on a code-switching protocol and is well-suited for linguistic and acoustic-phonetic studies. |
Strategies and Challenges for Crowdsourcing Regional Dialect Perception Data for Swiss German and Swiss French (L18-1)
Copied to clipboard
| Challenge: | a crowdsourcing project in the field of Swiss German dialects and Swiss French accents collects linguistic data. |
| Approach: | a gamified crowdsourcing platform was set up to collect linguistic data on Swiss German and Swiss French accents. |
| Outcome: | a gamified crowdsourcing platform collects linguistic data on Swiss German and Swiss French accents . the platform has provided 470,000 localizations, with 7,500 registered users and 30,000 anonymous visitors . |
PyramidCodec: Hierarchical Codec for Long-form Music Generation in Audio Domain (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to generate long music are inefficient and lack of structured representation. |
| Approach: | They propose a hierarchical discrete representation of audio for long audio-domain music generation using residual vector quantization on different levels of features. |
| Outcome: | The proposed method achieves competitive performance in terms of reconstruction quality and token per second (TPS) the proposed method facilitates training a language model that can generate well-structured long-form music for up to 3 minutes. |
Love Me, Love Me, Say (and Write!) that You Love Me: Enriching the WASABI Song Corpus with Lyrics Annotations (2020.lrec-1)
Copied to clipboard
| Challenge: | a corpus of songs enriched with metadata extracted from music databases on the Web contains 1.73M songs with lyrics (1.41M unique lyrics) a researcher proposes methods to extract relevant information from lyrics, including their structure segmentation, topic, explicitness of lyrics content, salient passages of a song and emotions conveyed. |
| Approach: | They propose to extract relevant information from lyrics by using music databases . they propose to use metadata extracted from music databases to analyze lyrics . |
| Outcome: | The proposed methods can be exploited by music search engines and music professionals to better handle large collections of lyrics. |
Investigating the Emergent Audio Classification Ability of ASR Foundation Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Text and vision foundation models can perform many tasks in a zero-shot setting . however, there has been little work on the zero-shoot abilities of ASR foundation models . |
| Approach: | They investigate the ability of ASR foundation models to perform zero-shot audio classification using text prompts and a decoding probability generator. |
| Outcome: | The proposed model outperforms state-of-the-art models on audio classification datasets without training them on extra data or adding any parameters. |
CONDUCT: An Expressive Conducting Gesture Dataset for Sound Control (L18-1)
Copied to clipboard
| Challenge: | Recent studies on music-gesture relationship focus on sound variations and expressiveness of gestures. |
| Approach: | They propose to use a database to create a set of expressive gestures for orchestral conductors . they assume that the gestures convey some meaning shared by most conductor . |
| Outcome: | The proposed database will be used to train a gesture recognition system for live sound control and modulation. |
Acoustic-to-Word Models with Conversational Context Information (N19-1)
Copied to clipboard
| Challenge: | Existing speech recognition models are built at a sentence level, and therefore it may not capture conversational context information. |
| Approach: | They propose a direct acoustic-to-word, end-to end speech recognition model that integrates a conversational context with other available information and directly recognizes words from speech. |
| Outcome: | The proposed model outperforms a standard end-to-end speech recognition system on the Switchboard conversational speech corpus and shows that it is more accurate than existing models. |
Fluent Translations from Disfluent Speech in End-to-End Speech Translation (N19-1)
Copied to clipboard
| Challenge: | Disfluency removal is an intermediate step between speech recognition and machine translation (MT) with the rise of end-to-end speech translation systems, disfluency recognition and removal needs to be incorporated into the model architectures or handled as a post-processing step. |
| Approach: | They propose to use a sequence-to-sequence model to translate from noisy, disfluent speech to fluent text with disfluencies removed using the recently collected ‘copy-edited’ references for the Fisher Spanish-English dataset. |
| Outcome: | The proposed model generates fluent translations from disfluent speech using the recently collected ‘copy-edited’ references for the Fisher Spanish-English dataset. |
ConFiguRe: Exploring Discourse-level Chinese Figures of Speech (2022.coling-1)
Copied to clipboard
| Challenge: | Figures of speech often deviate from their literal meanings to express deeper semantic implications. |
| Approach: | They propose a concept of figurative unit, which is the carrier of a figure, and build a Chinese corpus for Contextualized Figure Recognition. |
| Outcome: | The proposed model is based on 12 types of figures commonly used in Chinese . it shows that the proposed tasks are challenging for existing models . |
A Comprehensive Evaluation of Incremental Speech Recognition and Diarization for Conversational AI (2020.coling-main)
Copied to clipboard
| Challenge: | Automatic Speech Recognition (ASR) systems are increasingly powerful and more numerous with several options existing as a service. |
| Approach: | They evaluate the most popular automatic speech recognition systems with metrics and experiments designed with these standards in mind. |
| Outcome: | The most popular ASR systems are Microsoft and IBM, and none are suitable for natural spontaneous conversations in real-time. |
Mutual-Learning Improves End-to-End Speech Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to end-to-end speech translation (E2E) models only allow one way knowledge transfer, which is limited by the performance of the teacher model. |
| Approach: | They propose a one-way knowledge transfer paradigm where the MT and ST models are collaboratively trained and considered as peers rather than teacher/student. |
| Outcome: | The proposed model improves the performance of end-to-end speech translation (ST) task by combining knowledge from two models with peer models. |
Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task (2021.acl-long)
Copied to clipboard
| Challenge: | Pretraining and multitask learning are widely used to improve the speech translation performance. |
| Approach: | They propose to train a speech translation model along with an auxiliary text translation task. |
| Outcome: | The proposed method improves translation quality by more than 2 BLEU over a strong baseline and achieves state-of-the-art results on the MuST-C English-German, English-French and English-Spanish language pairs. |
Methods of Automatic Matrix Language Determination for Code-Switched Speech (2024.emnlp-main)
Copied to clipboard
| Challenge: | Code-switching (CS) is the process of speakers switching between two or more languages in spoken or written language. |
| Approach: | They propose to use the Matrix Language Frame theory to describe CS speech . they compare MLID of English/Mandarin and English/Spanish CS to acoustic language identity . |
| Outcome: | The proposed models outperform monolingual models in acoustic language identity recognition tasks. |
Multitask Learning for Grapheme-to-Phoneme Conversion of Anglicisms in German Speech Recognition (2022.lrec-1)
Copied to clipboard
| Challenge: | Anglicisms are a challenge in German speech recognition due to their irregular pronunciation compared to native German words. |
| Approach: | They propose a multitask sequence-to-sequence approach for grapheme-tophoneme conversion to improve the phonetization of Anglicisms. |
| Outcome: | The proposed model reduces the word error rate by 1 % and the Anglicism error rate, while still maintaining the accuracy of the baseline model. |
Overlaps and Gender Analysis in the Context of Broadcast Media (2022.lrec-1)
Copied to clipboard
| Challenge: | Using gender and overlap annotations, we characterise interactions between speakers according to their gender and role in broadcast media. |
| Approach: | They propose to characterise interactions between speakers according to their gender and role in broadcast media by using a small dataset of 93 recordings from LCP French channel. |
| Outcome: | The proposed method could improve the efficiency of qualitative studies conducted in human sciences. |
Towards Building an Automatic Transcription System for Language Documentation: Experiences from Muyu (2020.lrec-1)
Copied to clipboard
| Challenge: | Language documentation is a rapidly growing field due to its urgency. |
| Approach: | They propose to use phoneme recognition to automatically recognize spoken languages and translate them to global languages. |
| Outcome: | The proposed tool performs better than existing methods with American English, Austrian German and Slovenian as source and target languages. |
Learning Paralinguistic Features from Audiobooks through Style Voice Conversion (2021.naacl-main)
Copied to clipboard
| Challenge: | Paralinguistics, the non-lexical components of speech, play a crucial role in human-human interaction. |
| Approach: | They propose a framework that enables a neural network to learn to extract paralinguistic attributes from speech using data that are not annotated for emotion. |
| Outcome: | The proposed framework improves on emotion recognition and speaking style detection tasks. |
Analyzing analytical methods: The case of phonology in neural models of spoken language (2020.acl-main)
Copied to clipboard
| Challenge: | Recent studies have focused on the strengths and weaknesses of various methods for analyzing phonology representations. |
| Approach: | They propose to use diagnostic classifiers and representational similarity analysis to quantify to what extent phonemes and phoneme sequences are encoded. |
| Outcome: | The proposed method is based on two commonly applied techniques . it shows that global-scope methods yield more consistent and interpretable results . |
Modeling the Music Genre Perception across Language-Bound Cultures (2020.emnlp-main)
Copied to clipboard
| Challenge: | a prevalent approach to culturally study music genres assumes that the same music genre is associated with the items in all cultures. |
| Approach: | They propose to use distributed concept embeddings and ontologies to obtain cross-lingual music genre annotations using language-specific semantic representations. |
| Outcome: | The proposed model can be compared with existing models using domain-dependent cross-lingual corpus. |
Detecting Dementia from Long Neuropsychological Interviews (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies suggest examiner's language can influence cognitive impairment classifications. |
| Approach: | They propose a three-stage pipeline to detect dementia from exam recordings to mitigate the influence of the examiner on automatic dementia identification decisions. |
| Outcome: | The proposed pipeline mitigates the influence of the examiner on automatic dementia identification decisions in real-world neuropsychological exams. |
WER We Stand: Benchmarking Urdu ASR Models (2025.coling-main)
Copied to clipboard
| Challenge: | This paper analyzes the performance of three ASR models for low-resource languages like Urdu . low-rural languages like urdu have significant gaps in accuracy and reliability . |
| Approach: | They evaluate the performance of three ASR models: Whisper, MMS, and Seamless-M4T . they present the first conversational speech dataset for benchmarking Urdu ASR systems . |
| Outcome: | The proposed model families outperform Whisper, MMS, and Seamless-M4T on two types of speech datasets. |
Beyond Common Words: Enhancing ASR Cross-Lingual Proper Noun Recognition Using Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | In this work, we address the challenge of cross-lingual proper noun recognition in automatic speech recognition systems where proper nodes in an utterance may originate from a language different from the language in which the ASR system is trained. |
| Approach: | They propose a dictionary-based method to correct ASR predictions in a large language model . |
| Outcome: | The proposed method significantly reduces word error rates across cross-lingual proper noun recognition tasks involving three secondary languages. |
DECM: Evaluating Bilingual ASR Performance on a Code-switching/mixing Benchmark (2024.lrec-main)
Copied to clipboard
| Challenge: | Code-switched (CSW) speech is a linguistic phenomenon that occurs when spoken utterances switch languages between sentences. |
| Approach: | They propose to use a dataset to evaluate German-English CSW speech . they show that the dataset includes splits with varying degrees of CSW . |
| Outcome: | The proposed dataset includes spontaneous speech from diverse domains, enabling realistic CSW evaluation in German-English. |
STTATTS: Unified Speech-To-Text And Text-To-Speech Model (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a multi-task learning approach is currently available for speech recognition and speech synthesis models . |
| Approach: | They propose a parameter-efficient approach to learning ASR and TTS jointly . they use English as a resource-rich language and Arabic as 'low-resource' language . |
| Outcome: | The proposed model saves 50% of computational and memory costs while learning ASR and TTS jointly. |
MuPe Life Stories Dataset: Spontaneous Speech in Brazilian Portuguese with a Case Study Evaluation on ASR Bias against Speakers Groups and Topic Modeling (2025.coling-main)
Copied to clipboard
Sidney Evaldo Leal, Arnaldo Candido Junior, Ricardo Marcacini, Edresson Casanova, Odilon Gonçalves, Anderson Silva Soares, Rodrigo Freitas Lima, Lucas Rafael Stefanel Gris, Sandra Aluísio
| Challenge: | Recent datasets for automatic speech recognition in Brazilian Portuguese lack diversity in terms of age groups, regional accents, and education levels. |
| Approach: | They propose to use a dataset to analyze the impact of ASR in Brazilian Portuguese (BP) they demonstrate that current models are biased regarding age, education, and regional accents. |
| Outcome: | The proposed dataset helps mitigate biases in current ASR models regarding education levels and age groups. |
Evaluating the Expressive Appropriateness of Speech in Rich Contexts (2026.acl-long)
Copied to clipboard
Tianrui Wang, Ziyang Ma, Yizhou Peng, Haoyu Wang, Zhikang Niu, Zikang Huang, Yihao Wu, Yi-Wen Chao, Yu Jiang, Yuheng Lu, Guanrou Yang, Xuanchen Li, Hexin Liu, Chunyu Qiang, Cheng Gong, Yifan Yang, Tianchi Liu, Junyu Wang, Nana Hou, Meng Ge, Fuming You, Yang Wei, Zhongqian Sun, Hu Haifeng, Xiaobao Wang, Eng Siong Chng, Xie Chen, Longbiao Wang, Jianwu Dang
| Challenge: | Existing methods for evaluating expressive speech focus on word accuracy, naturalness, signal quality, or emotional intensity at the utterance level. |
| Approach: | They propose a framework for Evaluating Expressive Appropriateness in speech that assesses whether a speech sample aligns with the underlying communicative intent implied by its discourse-level narrative context. |
| Outcome: | The proposed framework outperforms existing speech evaluation and analysis systems on a human-annotated test set. |
LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | ***LLaST*** is a framework for building high-performance Large Language model based Speech-to-text Translation systems. |
| Approach: | They propose a framework for building high-performance Large Language model based Speech-to-text Translation systems. |
| Outcome: | The proposed model outperforms the CoVoST-2 benchmark and showcases exceptional scaling capabilities powered by LLMs. |
Representation Purification for End-to-End Speech Translation (2025.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to enhance speech translation focus on enhancing knowledge transfer . factors in speech that are not relevant to translation content, such as timbre and rhythm, often limit the efficiency of knowledge transfer. |
| Approach: | They propose a framework that excludes content-agnostic perturbations from speech representations to mitigate their negative impact on ST. |
| Outcome: | The proposed framework significantly improves translation performance across all translation directions in three settings and achieves preeminent performance under a *transcript-free* setting. |
Afrispeech-Dialog: A Benchmark Dataset for Spontaneous English Conversations in Healthcare and Beyond (2025.naacl-long)
Copied to clipboard
Mardhiyah Sanni, Tassallah Abdullahi, Devendra Deepak Kayande, Emmanuel Ayodele, Naome A Etori, Michael Samwel Mollel, Moshood O. Yekini, Chibuzor Okocha, Lukman Enegi Ismaila, Folafunmi Omofoye, Boluwatife A. Adewale, Tobi Olatunji
| Challenge: | Afrispeech-Dialog is a benchmark dataset of 50 simulated medical and non-medical African-accented English conversations . a 10%+ performance degradation is found in ASR systems on long-form, accented speech . |
| Approach: | They propose to use a dataset to evaluate automatic speech recognition systems on African-accented conversations. |
| Outcome: | The proposed dataset compares state-of-the-art speech recognition systems on accented conversations with native accents and shows a 10%+ performance degradation. |
Semi-supervised Development of ASR Systems for Multilingual Code-switched Speech in Under-resourced Languages (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing models for code-switching between languages are under-resourced and limited by text and acoustic data. |
| Approach: | They propose to construct four separate bilingual automatic speech recognisers corresponding to four different language pairs between which speakers switch frequently. |
| Outcome: | The proposed models are compared with a non-batch-wise approach and show that they perform better when used with sparse training data. |
Pronunciation Variants and ASR of Colloquial Speech: A Case Study on Czech (L18-1)
Copied to clipboard
| Challenge: | a standard speech recognition system uses a pronunciation component that maps tokens in the transcripts to their phonetic representations. |
| Approach: | They propose to use a pronunciation dictionary to map tokens in speech transcripts to phonetic representations. |
| Outcome: | The proposed pronunciation dictionary performs better than a standard rule-based pronunciation component. |
Mind the Gap: Static and Interactive Evaluations of Large Audio Models (2025.acl-long)
Copied to clipboard
Minzhi Li, William Barr Held, Michael J Ryan, Kunat Pipatanakul, Potsawee Manakul, Hao Zhu, Diyi Yang
| Challenge: | Recent work has focused on evaluating large audio models (LAMs) that directly accept audio inputs. |
| Approach: | They propose an interactive approach to evaluate large audio models and collect 7,500 LAM interactions from 484 participants. |
| Outcome: | The proposed model is based on a set of user-generated audio interfaces with 7,500 interactions from 484 participants. |
Non-Autoregressive Chinese ASR Error Correction with Phonological Training (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to correct ASR errors focus on fixed-length corrections, but rarely consider variable-length ones. |
| Approach: | They propose a non-autoregressive method to correct Chinese ASR errors . they use phonological tokens to extend the source sentence for variable-length correction . |
| Outcome: | The proposed method improves word error rate and speeds up inference by 6.2 times compared with the autoregressive model. |
Toward Joint Language Modeling for Speech Units and Text (2023.findings-emnlp)
Copied to clipboard
Ju-Chieh Chou, Chung-Ming Chien, Wei-Ning Hsu, Karen Livescu, Arun Babu, Alexis Conneau, Alexei Baevski, Michael Auli
| Challenge: | Speech and text are two major forms of human language and little effort has been made to model them together. |
| Approach: | They propose to combine speech and text models to create mixed speech-text data by using different tokenizers and automatic metrics to evaluate how well the model mixes speech and texts. |
| Outcome: | The proposed model improves over a speech-only baseline and shows zero-shot cross-modal transferability. |
DUB: Discrete Unit Back-translation for Speech Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | Discrete unit back-translation (DUB) is a back-translated speech-to-text translation (ST) technique that can be applied to ST . a modality gap between speech and text makes it difficult to transfer these techniques to ST due to the modality of the speech-text model. |
| Approach: | They propose a method to represent speech with discrete units instead of continuous features in direct ST. |
| Outcome: | The proposed method achieves comparable performance to existing methods that rely on large-scale external data. |
The Nautilus Speaker Characterization Corpus: Speech Recordings and Labels of Speaker Characteristics and Voice Descriptions (L18-1)
Copied to clipboard
| Challenge: | The Nautilus Speaker Characterization corpus is a conversational microphone speech recording corpus from 300 speakers. |
| Approach: | They present a speaker characterization corpus from 300 german speakers . they use four scripted and four semi-spontaneous dialogs to simulate telephone calls . |
| Outcome: | The speaker characterization corpus is presented in the acoustically-isolated room Nautilus . it comprises conversational microphone speech recordings from 300 speakers . the data will be made freely available to the scientific community . |
End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translation (2023.emnlp-main)
Copied to clipboard
Juan Pablo Zuluaga-Gomez, Zhaocheng Huang, Xing Niu, Rohit Paturi, Sundararajan Srinivasan, Prashant Mathur, Brian Thompson, Marcello Federico
| Challenge: | Conventional speech-to-text translation systems are trained on single-speaker utterances, but they may not be applicable to real-life scenarios where the audio contains conversations by multiple speakers. |
| Approach: | They propose a speaker-turn-aware conversational speech translation model that integrates automatic speech recognition, speech translation and speaker turn detection using special tokens in a serialized labeling format. |
| Outcome: | The proposed model outperforms the reference systems on the multi-speaker condition while attaining comparable performance on the single-speakspeaker conditions. |
Evaluation of Automatic Formant Trackers (L18-1)
Copied to clipboard
| Challenge: | Formant trackers are widely used by speech scientists and speech engineers. |
| Approach: | They propose to use four open source formant trackers to evaluate the quality of speech recognition algorithms on the same American English data set. |
| Outcome: | The proposed formant trackers outperform LPC-based and Deep Learning on the American English data set VTR-TIMIT. |
German Radio Interviews: The GRAIN Release of the SFB732 Silver Standard Collection (L18-1)
Copied to clipboard
Katrin Schweitzer, Kerstin Eckart, Markus Gärtner, Agnieszka Falenska, Arndt Riester, Ina Rösiger, Antje Schweitzer, Sabrina Stehwien, Jonas Kuhn
| Challenge: | GRAIN contains German radio interviews and is annotated on multiple linguistic layers. |
| Approach: | They present GRAIN as part of the SFB732 Silver Standard Collection . GRAIN contains German radio interviews and is annotated on multiple linguistic layers . |
| Outcome: | The GRAIN data set contains German radio interviews and is annotated on multiple linguistic layers. |
ManaTTS Persian: a recipe for creating TTS datasets for lower resource languages (2025.naacl-long)
Copied to clipboard
| Challenge: | a new text-to-speech system is needed for visual impairments and the visually impaired . a text-based system is not available for all users, and is therefore limited to a limited audience. |
| Approach: | They propose to use ManaTTS, the most extensive publicly accessible Persian corpus . they use a fully transparent, MIT-licensed pipeline to collect transcribed speech datasets . |
| Outcome: | The proposed framework is the most extensive publicly accessible single-speaker Persian corpus . it includes tools for sentence tokenization, bounded audio segmentation, and forced alignment method . |
Automatic Identification of Code-Switching Functions in Speech Transcripts (2023.findings-acl)
Copied to clipboard
| Challenge: | Code-switching, or switching between languages, occurs for many reasons and has important linguistic, sociological, and cultural implications. |
| Approach: | They build a system to identify a wide range of functions for which speakers code-switch in everyday speech with an accuracy of 75% . they use a dataset of Hindi-English code-witched data to analyze their results . |
| Outcome: | The proposed system can identify a wide range of functions for which speakers code-switch in everyday speech, with an accuracy of 75% across all functions. |
Does Your Voice Assistant Remember? Analyzing Conversational Context Recall and Utilization in Voice Interaction Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in multi-turn voice interaction models have improved user-model communication, but whether open-source models share this ability remains unexplored. |
| Approach: | They propose to use ContextDialog to evaluate open-source interaction models' ability to recall past utterances to identify key limitations. |
| Outcome: | The proposed model retains and recalls past utterances better than closed-source models, but still struggles with questions about past . findings highlight key limitations in open-source model and suggest ways to improve memory retention and retrieval robustness. |
Towards a music-language mapping (L18-1)
Copied to clipboard
| Challenge: | a novel research idea investigates the possibility of musical input to speech interaction systems. |
| Approach: | They propose a musical language processing idea that investigates the possibility of musical input to speech interaction systems. |
| Outcome: | The proposed method could be used to map musical pieces and dialogues based on frequency of musical patterns . the proposed method is universal among different languages and easy to learn for musicians . |
End-to-End Simultaneous Speech Translation with Differentiable Segmentation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to perform simultaneous speech translation always separate segmentation from the underlying model. |
| Approach: | They propose to use Differentiable Segmentation (DiSeg) to learn segmentation from the translation model. |
| Outcome: | Experimental results show that the proposed model can learn segmentation from the translation model. |
Disfluency Detection using Auto-Correlational Neural Networks (D18-1)
Copied to clipboard
| Challenge: | a recent study proposes an auto-correlational neural network (ACNN) that can detect disfluency in speech . the model uses a convolutional neural system and augments it with a new auto-corrector . |
| Approach: | They propose a convolutional neural network model that captures "rough copy" dependencies . the model is based on a new auto-correlation operator that capture the kinds of "rough copies" dependency . |
| Outcome: | The proposed model outperforms the baseline CNN on a disfluency detection task with a 5% increase in f-score. |
Improved Transcription and Indexing of Oral History Interviews for Digital Humanities Research (L18-1)
Copied to clipboard
| Challenge: | Existing methods to improve transcription and indexing quality of Oral History interviews are not available. |
| Approach: | They propose to use a German Oral History test-set to improve transcription and indexing quality . they propose to combine acoustic modeling techniques with sophisticated neural networks . |
| Outcome: | The proposed system reduces word error rate by 28.3% on German Oral History test-set compared to baseline system . the Fraunhofer IAIS Audio Mining system can process long audio-files to automatically create time-aligned transcriptions. |
Open ASR for Icelandic: Resources and a Baseline System (L18-1)
Copied to clipboard
| Challenge: | Existing language resources are not sufficient for less-resourced languages, but a system with sufficient resources is needed. |
| Approach: | They describe available language resources and their preparation for use in a large vocabulary speech recognition system for Icelandic. |
| Outcome: | The proposed system improves on acoustic training sets and a speech corpus with a pronunciation dictionary. |
Automatically Identifying Complaints in Social Media (P19-1)
Copied to clipboard
| Challenge: | Complaining is a basic speech act used to express a negative mismatch between reality and expectations in a particular situation. |
| Approach: | They present a systematic analysis of complaints in computational linguistics . they collect annotated data set of written complaints expressed on Twitter . |
| Outcome: | The proposed model achieves predictive performance of up to 79 F1 using distant supervision. |
Speech Rate Calculations with Short Utterances: A Study from a Speech-to-Speech, Machine Translation Mediated Map Task (L18-1)
Copied to clipboard
| Challenge: | Computer mediated multi-lingual communication is becoming more frequent. |
| Approach: | They propose a method to verify if an utterance within a corpus is pronounced at a fast or slow pace. |
| Outcome: | The proposed method provides a value for the utterance speech rate in a corpus of short utterations. |
SpiCE: A New Open-Access Corpus of Conversational Bilingual Speech in Cantonese and English (2020.lrec-1)
Copied to clipboard
| Challenge: | SpiCE is a corpus of conversational Cantonese-English bilingual speech recorded in Vancouver, Canada . the corpus includes high-quality recordings of 34 early bilinguals in both English and Cantoneses . |
| Approach: | They describe the design, collection, orthographic transcription, and phonetic annotation of SpiCE . the corpus includes high-quality recordings of 34 early bilinguals in both English and Cantonese . |
| Outcome: | The SpiCE corpus includes high-quality recordings of 34 early bilinguals in both English and Cantonese . the corpus will promote bilingualism research for a typologically distinct pair of languages . |
The Indigenous Languages Technology project at NRC Canada: An empowerment-oriented approach to developing language software (2020.coling-main)
Copied to clipboard
Roland Kuhn, Fineen Davis, Alain Désilets, Eric Joanis, Anna Kazantseva, Rebecca Knowles, Patrick Littell, Delaney Lothian, Aidan Pine, Caroline Running Wolf, Eddie Santos, Darlene Stewart, Gilles Boulianne, Vishwa Gupta, Brian Maracle Owennatékha, Akwiratékha’ Martin, Christopher Cox, Marie-Odile Junker, Olivia Sammons, Delasie Torkornoo, Nathan Thanyehténhas Brinklow, Sara Child, Benoît Farley, David Huggins-Daines, Daisy Rosenblum, Heather Souter
| Challenge: | This paper describes the first, three-year phase of a project at the National Research Council of Canada that is developing software to assist Indigenous communities in preserving their languages and extending their use. |
| Approach: | They describe the first phase of a project at the National Research Council of Canada that is developing software to assist Indigenous communities in preserving their languages. |
| Outcome: | The proposed software will help Indigenous communities preserve and revitalize their languages and extend their use. |
ArzEn: A Speech Corpus for Code-switched Egyptian Arabic-English (2020.lrec-1)
Copied to clipboard
| Challenge: | a corpus of Arabic-English code-switching (CS) spontaneous speech is collected in an Egyptian university soundproof room . the language in Egypt is rather complex and poses many challenges to natural language processing (NLP) |
| Approach: | They present an Egyptian Arabic-English code-switching (CS) spontaneous speech corpus. |
| Outcome: | The proposed corpus is designed to be used in automatic speech recognition systems . it provides a useful resource for analyzing the CS phenomenon from linguistic, sociological, and psychological perspectives. |
Evaluation Phonemic Transcription of Low-Resource Tonal Languages for Language Documentation (L18-1)
Copied to clipboard
| Challenge: | Language documentation involves recording the speech of native speakers. |
| Approach: | They propose to use a neural network architecture to model phonemes and tones versus modelling them separately. |
| Outcome: | The proposed method improves efficiency, minimizes typographical errors and maintains transcription faithfulness to acoustic signal while highlighting phonetic and phonemic facts for linguistic consideration. |
Researching Less-Resourced Languages – the DigiSami Corpus (L18-1)
Copied to clipboard
| Challenge: | DigiSami project aims to support research on endangered languages . it uses spoken corpus and speech technology for the Fenno-Ugric language North Sami . |
| Approach: | They describe the DigiSami project and its research results for the Fenno-Ugric language North Sami . they discuss ethical and privacy issues related to data collection for less-resourced languages and indigenous communities . |
| Outcome: | The DigiSami project focuses on spoken corpus collection and speech technology for the Fenno-Ugric language North Sami. |
[b] = [d] - [t] + [p]: Self-supervised Speech Models Discover Phonological Vector Arithmetic (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on how self-supervised speech models encode rich phonetic information have not explored how they are structured. |
| Approach: | They conduct a comprehensive analysis of the underlying structure of S3M representations with particular attention to phonological vectors. |
| Outcome: | The proposed model encodes phonologically interpretable and compositional vectors, demonstrating phonology vector arithmetic. |
Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speech (2021.emnlp-main)
Copied to clipboard
| Challenge: | Automatic Speech Recognition systems perform poorly on atypical speech and heavily accented speech. |
| Approach: | They add a residual adapter to the encoder layer to improve model adaptation . they show that the residual adapters update only a tiny fraction of the model parameters . |
| Outcome: | The proposed model fine-tuning improves performance on atypical and accented speech . the system can update only a tiny fraction of the model parameters . |
A Survey of Multilingual Models for Automatic Speech Recognition (2022.lrec-1)
Copied to clipboard
| Challenge: | Automatic Speech Recognition (ASR) systems have achieved human-like performance for a few languages, but the majority of the world’s languages do not have usable systems due to the lack of large speech datasets to train these models. |
| Approach: | They propose to use unlabeled speech data to build multilingual ASR models that can be used for improved performance on low-resource languages. |
| Outcome: | The proposed models can be used to improve performance on low-resource languages by using unlabeled speech data. |
Learning Adaptive Segmentation Policy for End-to-End Simultaneous Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to perform simultaneous speech-to-text translation ignore contextual information and suffer from low translation quality. |
| Approach: | They propose an adaptive segmentation policy for simultaneous speech-to-text translation . it learns to segment the source streaming speech into meaningful units . |
| Outcome: | The proposed method achieves a good accuracy-latency trade-off over state-of-the-art methods on English-German and Chinese-English. |
Learning the Beauty in Songs: Neural Singing Voice Beautifier (2022.acl-long)
Copied to clipboard
| Challenge: | Existing techniques for pitch correction are limited to intonation but ignore the overall aesthetic quality. |
| Approach: | They propose a novel time-warping approach for pitch correction to synchronize the amateur recording with the template pitch curve. |
| Outcome: | The proposed model improves intonation and vocal tone while keeping content and vocal timbre. |
Where are we in Named Entity Recognition from Speech? (2020.lrec-1)
Copied to clipboard
| Challenge: | Named entity recognition is usually made through a pipeline process that consists of processing audio and applying a NER to the audio outputs. |
| Approach: | They propose an original 3-pass approach and explore the capability of an E2E system to do structured NER. |
| Outcome: | The proposed system performs better than the current pipeline approach. |
SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities (2022.acl-long)
Copied to clipboard
Hsiang-Sheng Tsai, Heng-Jui Chang, Wen-Chin Huang, Zili Huang, Kushal Lakhotia, Shu-wen Yang, Shuyan Dong, Andy Liu, Cheng-I Lai, Jiatong Shi, Xuankai Chang, Phil Hall, Hsuan-Jui Chen, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, Hung-yi Lee
| Challenge: | Existing evaluation methods for transfer learning are limited in speech research . authors show that pre-trained models transfer well across multiple tasks . |
| Approach: | They propose a benchmark to evaluate pre-trained models by increasing task diversity and difficulty over SUPERB. |
| Outcome: | The proposed benchmark increases task diversity and difficulty over SUPERB-SG. |
Self-Supervised Singing Voice Pre-Training towards Speech-to-Singing Conversion (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on speech-to-singing voice conversion (STS) are limited by the scarcity of paired speech-song data and the suboptimal quality of outputs. |
| Approach: | They propose a self-supervised singing voice pre-training model that transforms a speech-to-singing voice into a paired singing voice. |
| Outcome: | The proposed model improves both STS and singing voice synthesis tasks by combining spoken language and a self-supervised singing voice pre-training model. |
Phonotomizer: A Compact, Unsupervised, Online Training Approach to Real-Time, Multilingual Phonetic Segmentation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to phonetic segmentation are hierarchical and end-to-end . many mistakes in final output stem from subtle segmenter perturbations . |
| Approach: | They propose a phonetic segmentation system that trains on raw sound files alone . it can modulate computational exactness and reduce acoustic model size, they argue . |
| Outcome: | The proposed method reduces the size of the acoustic model and training epochs. |
MELD-ST: An Emotion-aware Speech Translation Dataset (2024.findings-acl)
Copied to clipboard
Sirou Chen, Sakiko Yahata, Shuichiro Shimizu, Zhengdong Yang, Yihang Li, Chenhui Chu, Sadao Kurohashi
| Challenge: | Emotion plays a crucial role in human conversation. |
| Approach: | They present a MELD-ST dataset for the emotion-aware speech translation task . they show that fine-tuning with emotion labels can enhance translation performance . |
| Outcome: | The proposed dataset shows that fine tuning with emotion labels can improve translation performance in some settings. |
What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing text normalization routines that target Indic scripts are flawed when applied to multilingual automatic speech recognition models. |
| Approach: | They propose to develop text normalization routines that leverage native linguistic expertise to ensure more robust and accurate evaluations of multilingual automatic speech recognition models. |
| Outcome: | The proposed normalization routines can be leveraged to improve performance metrics for Indic languages. |
Extracting Biomedical Entities from Noisy Audio Transcripts (2024.lrec-main)
Copied to clipboard
Nima Ebadi, Kellen Morgan, Adrian Tan, Billy Linares, Sheri Osborn, Emma Majors, Jeremy Davis, Anthony Rios
| Challenge: | Named Entity Recognition (NER) is particularly affected by noise, often termed the ASR-NLP gap. |
| Approach: | They propose a dataset to bridge the ASR-NLP gap in the biomedical domain by extracting adverse drug reactions and mentions of entities from the Brief Test of Adult Cognition by Telephone (BTACT) exam. |
| Outcome: | The proposed method can clean 2,000 clean and noisy recordings and eliminate errors using zero-shot and few-shot methods. |
FalAI: A Dataset for End-to-end Spoken Language Understanding in a Low-Resource Scenario (2024.lrec-main)
Copied to clipboard
Andres Pineiro-Martin, Carmen Garcia-Mateo, Laura Docio-Fernandez, Maria del Carmen Lopez-Perez, Jose Gandarela-Rodriguez
| Challenge: | End-to-end (E2E) Spoken Language Understanding systems extract structured information from speech signals using a single model. |
| Approach: | They propose to use a dataset to extract structured information from speech signals . they define splits for noisy audio, hesitant audio and audio where sentence has changed . |
| Outcome: | The proposed model exploits acoustic information and avoids cascading errors . falAI dataset is the largest public SLU dataset in Galician and first to be obtained in low-resource scenario. |
Improving Code-switched ASR with Linguistic Information (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies on code-switching have been limited to the individual languages, but the results are promising. |
| Approach: | They propose to apply linguistic theories to generate more realistic code-switching text, which is needed for language modelling in ASR. |
| Outcome: | The proposed system improves 2% on English-Spanish code-switching . Equivalence Constraint theory and part-of-speech labelling are particularly helpful for text generation and bring 2% improvement to ASR performance. |
Unveiling the Role of Pretraining in Direct Speech Translation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to train direct speech-to-text translation systems are pretraining the encoder on automatic speech recognition, thus losing efficiency in the training process. |
| Approach: | They propose to change the decoder cross-attention to integrate source information from earlier steps in training. |
| Outcome: | The proposed model can achieve comparable performance to the pretrained model while reducing training time. |
The role of context in neural pitch accent detection in English (2020.emnlp-main)
Copied to clipboard
| Challenge: | Prosody is a rich information source in natural language, serving as a marker for phenomena such as contrast. |
| Approach: | They propose a model that uses full utterances as input and adds an LSTM layer to detect prosodic events in speech. |
| Outcome: | The proposed model improves on the American English speech in the Boston University Radio News Corpus. |
Evaluation of Off-the-shelf Speech Recognizers on Different Accents in a Dialogue Domain (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing automatic speech recognition systems for non-American accents have a much higher error rate than for general american accents. |
| Approach: | They evaluate automatic speech recognition systems on agent-directed speech . they find that the performance is worse for non-American accents than for General American . |
| Outcome: | The ASR systems perform worse for non-American accents than for General American accents . the results suggest that training on non-native English speakers is needed to narrow the performance gap. |
The Distribution and Prosodic Realization of Verb Forms in German Infant-Directed Speech (L18-1)
Copied to clipboard
| Challenge: | Infant-directed speech is often seen as a predictor for infants' speech processing abilities, for instance speech segmentation or word learning. |
| Approach: | They examine the syntactic distribution, accentuation and prosodic phrasing of German verb forms and show that many verb forms are prime candidates for early segmentation. |
| Outcome: | The findings suggest that infants ought to be able to extract verbs as early as nouns, given appropriate stimulus materials. |
Developing Resources for Automated Speech Processing of Quebec French (2020.lrec-1)
Copied to clipboard
| Challenge: | acoustic models for automatic segmentation of Quebec French are not available for all languages . linguistic resources are developed to perform phonetic annotations in Quebec French . physical characteristics of speech can be observed in the production of sounds . |
| Approach: | They propose to use a French lexicon to train automatic QF segmentation models . they adapt existing pronunciation dictionary and acoustic model from existing ones . |
| Outcome: | The proposed tools perform the full process of speech segmentation in Quebec French. |
Speech Translation and the End-to-End Promise: Taking Stock of Where We Are (2020.acl-main)
Copied to clipboard
| Challenge: | Until recently, the only feasible approach to translating acoustic speech signals into text was the cascaded approach. |
| Approach: | They propose a classification of the main challenges of traditional approaches to speech translation . they argue that end-to-end models fall short due to compromises made to address data scarcity . |
| Outcome: | This paper provides a brief survey of the main challenges of traditional approaches in speech translation . it reveals that many end-to-end models fail due to compromises made to address data scarcity. |
Carcinologic Speech Severity Index Project: A Database of Speech Disorder Productions to Assess Quality of Life Related to Speech After Cancer (L18-1)
Copied to clipboard
Corine Astésano, Mathieu Balaguer, Jérôme Farinas, Corinne Fredouille, Pascal Gaillard, Alain Ghio, Imed Laaridh, Muriel Lalain, Benoît Lepage, Julie Mauclair, Olivier Nocaudie, Julien Pinquier, Oriol Pont, Gilles Pouchoulin, Michèle Puech, Danièle Robert, Etienne Sicard, Virginie Woisard
| Challenge: | Increasing mortality in cancerology highlights the importance of reducing the impact on the Quality of Life after cancer treatment. |
| Approach: | They collect a large database of french speech recordings aimed at validating Disorder Severity Indexes. |
| Outcome: | The collected data will be available to the scientific community through the GIS Parolotheque. |
SynPaFlex-Corpus: An Expressive French Audiobooks Corpus dedicated to expressive speech synthesis. (L18-1)
Copied to clipboard
| Challenge: | a French audiobooks corpus contains 87 hours of good audio quality speech . audiobooks provide mono-genre and multi-speaker speech whereas audiobooks usually provide a few hours of mono- and multispeakers . |
| Approach: | They present an expressive French audiobooks corpus containing eighty seven hours of speech . the corpus is annotated automatically and provides information as phone labels, phone boundaries, syllables, words or morpho-syntactic tagging. |
| Outcome: | The proposed corpus contains 87 hours of speech recorded by a single speaker . the data will allow developing models to better control expressiveness in speech synthesis . |
Bringing Order to Chaos: A Non-Sequential Approach for Browsing Large Sets of Found Audio Data (L18-1)
Copied to clipboard
| Challenge: | a new approach to search for sound in large archives is being developed . speech and speech technology researchers struggle to access large amounts of data . |
| Approach: | They propose a method for fast and efficient non-sequential browsing of sound in large archives that we know little about . they combine audio browsing through massively multi-object sound environments and an unsupervised dimensionality reduction algorithm to search for sound in public archives. |
| Outcome: | The proposed method is shown to combine well, resulting in rapid and interpretable observations. |
GIL-GALaD: Gender Inclusive Language - German Auto-Assembled Large Database (2024.lrec-main)
Copied to clipboard
| Challenge: | grammatically gendered languages such as German pose unique challenges in generating gender-inclusive language for corrective model training or fine-tuning. |
| Approach: | a corpus of German gender-inclusive language is assembled to help improve model training . grammatically gendered languages such as german pose unique challenges . authors describe most common strategies for gender- inclusive language in german . |
| Outcome: | a corpus of German gender-inclusive language is assembled and will be included in the release. |
SPADE: A Big Five-Mturk Dataset of Argumentative Speech Enriched with Socio-Demographics for Personality Detection (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent efforts to create such datasets from social media do not include continuous and contextualized language use. |
| Approach: | They propose to use argumentative speech to generate a dataset with continuous arguments labeled with the Big Five personality traits and enriched with socio-demographic data. |
| Outcome: | The proposed model leverages 436 (psycho)linguistic features extracted from transcribed speech and speaker-level metainformation with transformers to investigate which types of features contribute to the prediction of individual personality traits. |
Progress in Multilingual Speech Recognition for Low Resource Languages Kurmanji Kurdish, Cree and Inuktut (2022.lrec-1)
Copied to clipboard
| Challenge: | Using acoustic data, we develop automatic speech recognition systems for three low resource languages. |
| Approach: | They develop automatic speech recognition systems for three low resource languages using acoustic training data from 12 different languages in the hybrid DNN/HMM framework. |
| Outcome: | The proposed models are for three low resource languages: Kurmanji Kurdish, Cree and Inuktut. |
How to Do Politics with Words: Investigating Speech Acts in Parliamentary Debates (2024.lrec-main)
Copied to clipboard
| Challenge: | a new perspective on framing through the lens of speech acts investigates how politicians make use of different pragmatic speech act functions in political debates. |
| Approach: | They propose a new framework for framing through the lens of speech acts and an annotation scheme for political debates. |
| Outcome: | The proposed framework can predict speech acts with an avg. F1 of around 82.0% . the proposed framework is based on a dataset of German parliamentary debates . |
Language of Bargaining (2023.acl-long)
Copied to clipboard
| Challenge: | a new dataset is being developed to study how language shapes bilateral bargaining . a recent study examined the use of language in negotiation education . |
| Approach: | They propose a dataset to study how language shapes bilateral bargaining . they recruit participants via behavioral labs instead of crowdsourcing platforms . |
| Outcome: | The proposed dataset is based on an exercise in negotiation education . it shows that when subjects can talk, negotiations finish faster and prices drop . |
Annotation of Valence Unfolding in Spoken Personal Narratives (2022.lrec-1)
Copied to clipboard
Aniruddha Tammewar, Franziska Braun, Gabriel Roccabruna, Sebastian Bayerl, Korbinian Riedhammer, Giuseppe Riccardi
| Challenge: | Personal Narrative (PN) is the recollection of individuals’ life experiences, events, and thoughts along with the associated emotions in the form of a story. |
| Approach: | They annotate a corpus of spoken personal narratives with the emotion valence using discrete values and use a 5-point bipolar scale to measure their agreement. |
| Outcome: | The annotators annotate a corpus of spoken personal narratives with the emotion valence using discrete values on a 5-point bipolar scale ranging from -2 to +2 (0 for neutral). |
Textless Speech Emotion Conversion using Discrete & Decomposed Representations (2022.emnlp-main)
Copied to clipboard
Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu Anh Nguyen, Morgan Rivière, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, Yossi Adi
| Challenge: | Existing methods for modifying emotion of speech are difficult because emotion affects all levels simultaneously. |
| Approach: | They propose a method to convert a spoken language speech into a model of emotion . they use phonetic-content units, prosodic features, speaker, and emotion to modify the emotion a speech utterance has. |
| Outcome: | The proposed method beats text-based systems in terms of perceived emotion and audio quality. |
Burmese Speech Corpus, Finite-State Text Normalization and Pronunciation Grammars with an Application to Text-to-Speech (2020.lrec-1)
Copied to clipboard
Yin May Oo, Theeraphol Wattanavekin, Chenfang Li, Pasindu De Silva, Supheakmungkol Sarin, Knot Pipatsrisawat, Martin Jansche, Oddur Kjartansson, Alexander Gutkin
| Challenge: | Using crowd-sourced speech corpus and finite-state transducer grammars, we build a text-to-speech system for Burmese, a tonal Southeast Asian language from the Sino-Tibetan family. |
| Approach: | They propose an open-source crowd-sourced multi-speaker speech corpus and finite-state grammars for performing grapheme-to-phoneme conversion for Burmese. |
| Outcome: | The proposed system performs well for Burmese in a low-resource setting. |
STARS: A Unified Framework for Singing Transcription, Alignment, and Refined Style Annotation (2025.findings-acl)
Copied to clipboard
Wenxiang Guo, Yu Zhang, Changhao Pan, Zhiyuan Zhu, Ruiqi Li, ZheTao Chen, Wenhao Xu, Fei Wu, Zhou Zhao
| Challenge: | Existing automated singing annotation (ASA) methods tackle isolated aspects of the annotation pipeline. |
| Approach: | They propose a framework that addresses transcription, alignment, and refined style annotations. |
| Outcome: | The proposed framework delivers comprehensive multi-level annotations encompassing: (1) precise phoneme-audio alignment, (2) robust note transcription and temporal localization, (3) expressive vocal technique identification, and (4) global stylistic characterization including emotion and pace. |
InaGVAD : A Challenging French TV and Radio Corpus Annotated for Speech Activity Detection and Speaker Gender Segmentation (2024.lrec-main)
Copied to clipboard
| Challenge: | InaGVAD is an audio corpus collected from 10 French radio and 18 TV channels categorized into 4 groups: generalist radio, music radio, news TV, and generalist TV. |
| Approach: | They propose to use an audio corpus from 10 French radio and 18 TV channels to represent the acoustic diversity of French audiovisual programs. |
| Outcome: | The proposed system is trained on a single hour of data and achieved competitive results. |
Automatic Period Segmentation of Oral French (2020.lrec-1)
Copied to clipboard
| Challenge: | Analor is a semi-automatic tool for speech segmentation in periods but it only takes into account prosodic characteristics of speech. |
| Approach: | They propose to use a Fribourg model of macro-syntax to detect periods in syntactic and prosodic terms to develop an automatic tool for automatic segmentation of linguistic units. |
| Outcome: | The proposed tool is compared with an existing tool Analor which divides speech into smaller segments and that CRF models detect larger segments rather than macro-syntactic periods. |
Urdu Pitch Accents and Intonation Patterns in Spontaneous Conversational Speech (2020.lrec-1)
Copied to clipboard
| Challenge: | Recent studies of Urdu intonation describe scripted and laboratory speech . |
| Approach: | They summarise Urdu pitch accents and their intonation patterns using a simplified version of the Rhythm and Pitch labelling system and a simple RAP system. |
| Outcome: | The analysis of a hand-labelled telephone conversation shows that low pitch accents play an important role in Urdu spontaneous speech. |
Evaluation of Off-the-shelf Speech Recognizers Across Diverse Dialogue Domains (2020.lrec-1)
Copied to clipboard
| Challenge: | a recent study evaluated off-the-shelf automatic speech recognition systems . current state-of-the art systems perform poorly in domains that require special vocabulary and language models . |
| Approach: | They evaluate off-the-shelf automatic speech recognition systems across different dialogue domains . they use data collected from deployed spoken dialogue systems and human-human conversations . |
| Outcome: | The evaluation is aimed at non-experts with limited experience in speech recognition . the results show that the performance of each speech recognizer can vary significantly depending on the domain . |
MaSS: A Large and Clean Multilingual Corpus of Sentence-aligned Spoken Utterances Extracted from the Bible (2020.lrec-1)
Copied to clipboard
| Challenge: | The Bible is the same for all the languages, thus constituting a multilingual and comparable 2 spoken corpus, is not exploited to date. |
| Approach: | They propose to add multilingual links between small speech segments in different languages . they use a large dataset of 8,130 parallel spoken utterances across 8 languages - maSS . |
| Outcome: | The proposed model can build automatic speech recognition models for 700 languages. |
The Objective and Subjective Sleepiness Voice Corpora (2020.lrec-1)
Copied to clipboard
| Challenge: | Following chronic sleep disorders involves multiple appointments between doctors and patients which often results in episodic follow-ups with unevenly spaced interviews. |
| Approach: | They propose to use a large database to assess the sleepiness level of highly phenotyped patients that complain from excessive daytime sleepiness instead of healthy subjects. |
| Outcome: | The proposed model is based on recordings from patients suffering from excessive daytime sleepiness instead of healthy subjects and incites them to sleep contrary to existing stressing sleepiness deprivation paradigms. |
Interpretable Assessment of Speech Intelligibility Using Deep Learning: A Case Study on Speech Disorders Due to Head and Neck Cancers (2024.lrec-main)
Copied to clipboard
Sondes Abderrazek, Corinne Fredouille, Alain Ghio, Muriel Lalain, Christine Meunier, Mathieu Balaguer, Virginie Woisard
| Challenge: | Using deep learning, speech disorders can be evaluated by perceptual measures, but they are subject to subjectivity and lack of reproducibility. |
| Approach: | They propose to use deep-learning to explain hidden representations in a deep- learning speech model to provide a deeper understanding of the final intelligibility assessment of patients with Head and Neck Cancers. |
| Outcome: | The proposed approach predicts speech intelligibility and severity of patients with Head and Neck Cancers while giving relevant interpretations of the final assessment at the phonemes and phonetic feature levels. |
On Construction of the ASR-oriented Indian English Pronunciation Dictionary (2020.lrec-1)
Copied to clipboard
| Challenge: | Indian English (IE) has distinctive characteristics, especially phonologically, from other varieties of English. |
| Approach: | They build a small IE spontaneous speech corpus and use a linguistically-guided IE pronunciation dictionary to apply it to IE. |
| Outcome: | The proposed system performs better on IE spontaneous speech data than the one trained with CMUdict. |
Call My Net 2: A New Resource for Speaker Recognition (2020.lrec-1)
Copied to clipboard
| Challenge: | Call My Net 2 (CMN2) corpus features Tunisian Arabic conversations between friends and family . call recordings include speech in various realistic and natural acoustic settings, both noisy and non-noisy. |
| Approach: | They introduce the Call My Net 2 (CMN2) corpus, a new resource for speaker recognition featuring Tunisian Arabic conversations between friends and family. |
| Outcome: | The Call My Net 2 (CMN2) corpus contains data from over 400 Tunisian Arabic speakers . each speaker made 10 or more calls each lasting up to 10 minutes . |
Implicit Memory Transformer for Computationally Efficient Simultaneous Speech Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | Simultaneous speech translation is an essential communication task difficult for humans whereby a translation is generated concurrently with oncoming speech inputs. |
| Approach: | They propose a transformer that implicitly retains memory through a new left context method, removing the need to explicitly represent memory with memory banks. |
| Outcome: | The proposed method provides a substantial speedup on the encoder forward pass with nearly identical translation quality when compared with the state-of-the-art approach that uses left context and memory banks. |
Audio-Based Linguistic Feature Extraction for Enhancing Multi-lingual and Low-Resource Text-to-Speech (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to synthesize speech for low-resource languages require a substantial amount of source language corpora to generate the linguistic knowledge that can be reused for speech synthesis. |
| Approach: | They propose a method that extracts linguistic features from audio input while effectively filtering out miscellaneous acoustic information including speaker-specific attributes like timbre. |
| Outcome: | The proposed method extracts linguistic features from audio input while effectively filtering out miscellaneous acoustic information including speaker-specific attributes like timbre. |
Killkan: The Automatic Speech Recognition Dataset for Kichwa with Morphosyntactic Information (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing datasets for automatic speech recognition (ASR) in the endangered Kichwa language have been limited. |
| Approach: | They present Killkan, the first dataset for automatic speech recognition (ASR) in the Kichwa language, an indigenous language of Ecuador. |
| Outcome: | The proposed dataset shows that it can be used to build an automatic speech recognition system for the endangered language with reliable quality despite its small size. |
Seshat: a Tool for Managing and Verifying Annotation Campaigns of Audio Data (2020.lrec-1)
Copied to clipboard
Hadrien Titeux, Rachid Riad, Xuan-Nga Cao, Nicolas Hamilakis, Kris Madden, Alejandrina Cristia, Anne-Catherine Bachoud-Lévi, Emmanuel Dupoux
| Challenge: | Seshat is a software for the automated management of annotation campaigns for audio/speech data. |
| Approach: | They propose a system for the automated management of annotation campaigns for audio/speech data which addresses these challenges. |
| Outcome: | The proposed system computes an associated inter-annotator agreement with the gamma measure taking into account the categorisation and segmentation discrepancies. |
Language and Speech Technology for Central Kurdish Varieties (2024.lrec-main)
Copied to clipboard
| Challenge: | a recent study focused on the Kurdish language, a less-resourced Indo-European language spoken by over 30 million speakers. |
| Approach: | They propose to develop resources for language and speech technology for Kurdish . they report the performance of machine translation, automatic speech recognition and language identification . |
| Outcome: | The proposed model is based on transcribing movies and TV series as an alternative to fieldwork. |
URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a lack of comprehensive evaluations for SDMs in speech-to-speech (S2S) scenarios is a major challenge for end-to end spoken dialogue models. |
| Approach: | They propose to provide an extensive evaluation framework for end-to-end spoken dialogue models (SDMs) that includes both cognitive dimensions and paralinguistic cues . |
| Outcome: | The proposed benchmark is divided into two difficulty levels: basic track and pro track, each comprising 20 test sets, evaluating the spoken dialogue model’s abilities in U**nderstanding, **R**easoning, and **O**ral conversation. |
Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Evaluating debate speeches requires a deep understanding of arguments at multiple levels. |
| Approach: | They propose a benchmark task for LLM judges based on annotated debate speeches . they analyze the judgment capabilities and behavior of frontier LLMs . |
| Outcome: | The proposed task requires a comprehensive understanding of argumentation and its arguments. |
Amplifying Trans and Nonbinary Voices: A Community-Centred Harm Taxonomy for LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies on harms of language technology to transgender and nonbinary people focus on misgendering and stereotyping . |
| Approach: | They propose a taxonomy of harms for large language models and heuristics for evaluation to help identify harmful behavior in LLMs. |
| Outcome: | The proposed model-based approach combines surveys and focus groups with community experts to identify harmful behavior in large language models. |
Hope vs. Hate: Understanding User Interactions with LGBTQ+ News Content in Mainstream US News Media through the Lens of Hope Speech (2025.emnlp-main)
Copied to clipboard
| Challenge: | a new study examines how users interact with LGBTQ+ news content . a corpus of 1,419,047 comments on 3,161 YouTube news videos is used to analyze the content - both positive and negative - of cable news outlets. |
| Approach: | They analyze how users interact with LGBTQ+ news content via a corpus of 1,419,047 comments on 3,161 YouTube news videos of major US cable news outlets. |
| Outcome: | The proposed classifier detects positive (hope speech), negative, neutral, and irrelevant content. |
Multilingual Turn-taking Prediction Using Voice Activity Projection (2024.lrec-main)
Copied to clipboard
| Challenge: | a monolingual model does not make good predictions when applied to other languages, but a multilingual model is able to discern the language of the input signal. |
| Approach: | They propose to use a multilingual voice activity projection model to predict voice activities of spoken dialogue participants in English, Mandarin, and Japanese data. |
| Outcome: | The proposed model predicts the upcoming voice activities of participants in dyadic dialogue on multilingual data, encompassing English, Mandarin, and Japanese. |
myMediCon: End-to-End Burmese Automatic Speech Recognition for Medical Conversations (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing medical conversation speech corpora for Burmese are limited, despite advances in ASR. |
| Approach: | They propose to use a manually curated medical conversation speech corpus for Burmese to examine the performance of ASR models. |
| Outcome: | The proposed model outperforms the Transformer model and the Recurrent Neural Network (RNN) models. |
nEMO: Dataset of Emotional Speech in Polish (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing datasets covering Slavic languages do not accurately represent basic emotional states. |
| Approach: | They propose to use a Polish corpus of emotional speech to represent basic emotional states. |
| Outcome: | The proposed corpus represents six emotional states in Polish, with 9 actors participating in the study. |
Out of the Mouths of MPs: Speaker Attribution in Parliamentary Debates (2024.lrec-main)
Copied to clipboard
| Challenge: | Identifying who says what to whom is an essential prerequisite for analysing human communication. |
| Approach: | They propose a new corpus for speaker attribution in german parliamentary debates . the data includes more than 7,700 manually annotated events of speech, thought and writing . they then apply their model to predict speech events in 20 years of debates and investigate the use of factives in the rhetoric of MPs. |
| Outcome: | The proposed model predicts speech events in 20 years of debates and investigates the use of factives in the rhetoric of MPs. |
Parameter-Efficient Transfer Learning for End-to-end Speech Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to improve end-to-end speech translation are limited by the availability of labeled data. |
| Approach: | They propose a method which utilizes two lightweight adaptation techniques to modulate Attention and the Feed-Forward Network while preserving the capabilities of pre-trained models. |
| Outcome: | The proposed method outperforms baseline models and significantly improves performance in low-resource settings. |
Muse: Towards Reproducible Long-Form Song Generation with Fine-Grained Style Control (2026.findings-acl)
Copied to clipboard
Changhao Jiang, Jiahao Chen, Zhenghao Xiang, Zhixiong Yang, Hanchen Wang, Jiabao Zhuang, Xinmeng Che, Jiajun Sun, Hui Li, Yifei Cao, Shihan Dou, Ming Zhang, Junjie Ye, Tao Ji, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Recent commercial systems such as Suno demonstrate strong capabilities in long-form song generation, but academic research remains non-reproducible due to the lack of publicly available training data. |
| Approach: | They propose a system for long-form song generation with fine-grained style conditioning that includes a licensed synthetic dataset and a song generation model, Muse. |
| Outcome: | The proposed system achieves competitive performance on phoneme error rate, text–music style similarity, and audio aesthetic quality while enabling controllable segment-level generation across different musical structures. |
C3: A Bilingual Benchmark for Spoken Dialogue Models Exploring Challenges in Complex Conversations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent developments in spoken dialogue models have created a gap in understanding their effectiveness in comprehending and emulating human conversations. |
| Approach: | They present a benchmark dataset which comprises 1,079 instances in English and Chinese to examine their effectiveness in emulating human conversations. |
| Outcome: | The proposed model outperforms existing models in English and Chinese by using an LLM-based evaluation method that closely aligns with human judgment. |
TokenVerse: Towards Unifying Speech and NLP Tasks via Transducer-based ASR (2024.emnlp-main)
Copied to clipboard
Shashi Kumar, Srikanth Madikeri, Juan Pablo Zuluaga Gomez, Iuliia Thorbecke, Esaú Villatoro-tello, Sergio Burdisso, Petr Motlicek, Karthik S, Aravind Ganapathiraju
| Challenge: | Existing approaches to automatic speech recognition use cascaded pipelines for tasks like voice activity detection, diarization, transcription and subsequent processing. |
| Approach: | They propose a single Transducer-based model that integrates task-specific tokens into the reference text during ASR model training, streamlining inference and eliminating the need for separate NLP models. |
| Outcome: | The proposed model outperforms the existing pipeline on speaker change detection, endpointing, and NER tasks while outperforming the existing model in individual task performance. |
Does Context Matter? A Prosodic Comparison of English and Spanish in Monolingual and Multilingual Discourse Settings (2025.emnlp-main)
Copied to clipboard
| Challenge: | a large number of studies on prosody in languages have focused on monolingual discourse contexts . a recent study focused on the prosodic features of monolingual speech in multilingual contexts. |
| Approach: | They compare prosody of monolingual English and Spanish in monolingual and multilingual settings . they find that monolingual speech produced in a monolingual context is prosodically different from that produced in multilingual context . |
| Outcome: | The proposed study is the first to incorporate multilingual discourse contexts into the study of native-level monolingual prosody. |
From perception to production: how acoustic invariance facilitates articulatory learning in a self-supervised vocal imitation model (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that map variable acoustic inputs into appropriate articulatory movements without explicit instruction are inadequate for infants. |
| Approach: | They propose a model that maps acoustic inputs into articulatory movements without explicit instruction for infants. |
| Outcome: | The proposed model outperforms MFCC features in both single- and multi-speaker settings and provides optimal representations for articulatory learning. |
SpeechQE: Estimating the Quality of Direct Speech Translation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in automatic quality estimation for machine translation focus on written language, leaving the speech modality underexplored. |
| Approach: | They propose a new quality estimation system based on cascaded and end-to-end architectures. |
| Outcome: | The proposed system is better suited to estimating the quality of direct speech translation than existing systems designed for text translation. |
A Unified Feature Mixture Framework for Joint Speech and Singing Deepfake Detection (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for deepfake detection fail under speech-to-singing domain shift . a speech-retentive multi-domain fine-tuning strategy enables adaptation to singing . |
| Approach: | They propose a unified deepfake detector based on a multi-branch mixture-of-experts architecture that integrates three complementary feature views. |
| Outcome: | The proposed detector achieves 1.82% EER on CtrSVDD, compared to 37–62% for existing detectors . it can generalize to unseen generators and preserve strong speech performance . |
What Do Neural Speech Models Know About Phonology? Evidence from Structured Phoneme Confusions (2026.findings-acl)
Copied to clipboard
| Challenge: | acoustic and phonological models of speech recognition are often limited to the phoneme level . a recent study has shown that phoneme confusions are strongly structured in phonology space . |
| Approach: | They adopt a featural representation of phonemes grounded in phonological theory which models speech sounds as structured bundles of distinctive articulatory and acoustic properties. |
| Outcome: | The proposed model allows us to analyse phoneme confusions at a finer granularity and to investigate whether certain phonological features are more vulnerable than others. |
Difference in Task Performance on Sparse Speech Representations (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for learning speech representations that are useful for a variety of downstream tasks have been extensively investigated in different domains. |
| Approach: | They propose to train Autoencoders with varying sparsity levels using three SSL features and evaluate them on six tasks of SUPERB: speech enhancement, speaker identification, speech Emotion Recognition, phone recognition, automatic speech recognition and slot filling. |
| Outcome: | The proposed model can be used to learn speech representations that are useful for a variety of downstream tasks. |
Sonos Voice Control Bias Assessment Dataset: A Methodology for Demographic Bias Assessment in Voice Assistants (2024.lrec-main)
Copied to clipboard
Chloe Sekkat, Fanny Leroy, Salima Mdhaffar, Blake Perry Smith, Yannick Estève, Joseph Dureau, Alice Coucke
| Challenge: | Recent studies show voice assistants do not perform equally well for everyone . however, research on demographic robustness of speech technologies is still scarce . |
| Approach: | They propose a statistical method to detect demographic bias using a large dataset with controlled demographic tags. |
| Outcome: | The proposed method shows statistically significant differences in performance across age, dialectal region and ethnicity. |
The Role of Creaky Voice in Turn Taking and the Perception of Speaker Stance: Experiments Using Controllable TTS (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in spontaneous text-to-speech (TTS) have enabled the realistic generation of creaky voice, a voice quality known for its diverse pragmatic and paralinguistic functions. |
| Approach: | They used a creaky voice detection tool and a neural TTS engine to control creaky phonation in a spontaneous speech corpus to investigate the effect of creaky voices on perceived certainty, valence, sarcasm, and turn finality. |
| Outcome: | The proposed model enables the realistic synthesis of creaky voice in perceptual tests without formal training. |
The Slovak Autistic and Non-Autistic Child Speech Corpus:Task-Oriented Child-Adult Interactions (2024.lrec-main)
Copied to clipboard
| Challenge: | Presented is the Slovak Autistic and Non-Autistic Child Speech Corpus . corpus contains over 15 hours of speech . |
| Approach: | They present a Slovak autistic and non-autistic child speech corpus . the corpus was primarily recorded to investigate lexical alignment . |
| Outcome: | The Slovak Autistic and Non-Autistic Child Speech Corpus contains over 15 hours of speech . the corpus can be shared with researchers and replicated in future research . |
The Swedish Parliament Corpus 1867 – 2022 (2024.lrec-main)
Copied to clipboard
Väinö Aleksi Yrjänäinen, Fredrik Mohammadi Norén, Robert Borges, Johan Jarlbrink, Lotta Åberg Brorsson, Anders P. Olsson, Pelle Snickars, Måns Magnusson
| Challenge: | The Swedish Parliament Corpus is a new research corpus for the Swedish parliament. |
| Approach: | They propose to expand the Swedish Parliament corpus by providing a database of all members of parliament over 150 years. |
| Outcome: | The new corpus facilitates detailed analysis of parliamentary speeches in several research fields. |
UniVocal: Unified Speech-Singing Code-Switching Synthesis (2026.acl-long)
Copied to clipboard
| Challenge: | Existing systems cannot automatically determine when to switch between modes based on text content. |
| Approach: | They propose a unified framework that implicitly infers vocal modes from text context to pioneer SCS Synthesis. |
| Outcome: | The proposed framework infers vocal modes solely from text context to pioneer SCS Synthesis. |
From Naturalness to Norms: Interactional Cultural Competence for SpeechLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Spoken language models are increasingly real-time conversational actors. |
| Approach: | They propose a speech-first view of cultural competence as interactional competence . they synthesize social-science foundations into a taxonomy of culture-bearing signals in speech . |
| Outcome: | The proposed model is based on a theory-derived taxonomy of culture-bearing signals in speech . it shows that cultural appropriateness is not a generic human-likeness . |
Why Voice Biomarkers of Psychiatric Disorders Are Not Used in Clinical Practice? Deconstructing the Myth of the Need for Objective Diagnosis (2024.lrec-main)
Copied to clipboard
| Challenge: | Anxiety and depression are the most prevalent mental disorders, affecting 3.9% and 3.6% of the world's population . |
| Approach: | They propose to shift the estimation of diagnoses towards estimation of clinical symptoms and signs, which address the limitations raised against diagnosis estimation. |
| Outcome: | The proposed paradigm shift will empower the use of vocal biomarkers in clinical practice. |
WoW-Bench: Evaluating Fine-Grained Acoustic Perception in Audio-Language Models via Marine Mammal Vocalizations (2026.findings-acl)
Copied to clipboard
| Challenge: | Large audio-language models extend language understanding into the auditory domain, yet their ability to perform low-level listening, such as pitch and duration detection, remains underexplored. |
| Approach: | They propose a global benchmark to evaluate low-level auditory perception and cognition using marine mammal vocalizations to better assess models’ low- level listening. |
| Outcome: | The proposed models show performance far below human levels, indicating a need for stronger auditory grounding in LALMs. |
XLSR-MamBo: Scaling the Hybrid Mamba-Attention Backbone for Audio Deepfake Detection (2026.findings-acl)
Copied to clipboard
| Challenge: | Advanced speech synthesis technologies have enabled highly realistic speech generation, posing security risks that motivate research into audio deepfake detection (ADD). |
| Approach: | They propose a modular framework that integrates an XLSR front-end with synergistic Mamba-Attention backbones to capture artifacts in spoofed speech signals. |
| Outcome: | The proposed framework achieves competitive performance on the ASVspoof 2021 LA, DF, and In-the-Wild benchmarks compared to other state-of-the art systems. |
S^4: Operationalizing Speech Act Theory for Strategic Semi-Structured Psychiatric Interview (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for psychiatric interviewing degenerate into rigid interrogation or aimless chitchat due to a lack of strategic planning. |
| Approach: | They propose a framework for psychiatric interviewing grounded in Speech Act Theory that integrates a large-scale dataset with fine-grained psychic speech act annotations. |
| Outcome: | The proposed framework outperforms baselines in psychiatric interviewing. |
TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding (2026.acl-long)
Copied to clipboard
| Challenge: | a critical ambiguity persists regarding what constitutes "joint ASR and diarization" a unified framework for multi-speaker ASR is proposed, but it is not yet clear what constitute "diarization." |
| Approach: | They propose a unified LLM-based framework that uses Temporal Anchor Grounding for joint multi-speaker ASR and diarization. |
| Outcome: | The proposed framework improves on AMI and AliMeeting benchmarks on speaker-content alignment . the proposed framework achieves consistent improvements in Diarization Error Rate over strong baselines . |
Prosody as Supervision: Bridging the Non-Verbal–Verbal for Multilingual Speech Emotion Recognition (2026.acl-long)
Copied to clipboard
| Challenge: | Existing paradigms for low-resource multilingual speech emotion recognition rely on labeled verbal speech and lack cross-lingual transfer. |
| Approach: | They propose a paralinguistic supervision paradigm for low-resource multilingual speech emotion recognition that leverages non-verbal vocalizations to exploit prosody-centric emotion cues. |
| Outcome: | The proposed framework outperforms Euclidean counter parts and strong SSL baselines in the language-based evaluation of low-resource multilingual speech emotion recognition (LRM-SER) |
Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback (2026.findings-acl)
Copied to clipboard
Siddhant Arora, Jinchuan Tian, Jiatong Shi, Hayato Futami, Yosuke Kashiwagi, Emiru Tsunoo, Shinji Watanabe
| Challenge: | Existing studies on reinforcement learning from human or AI feedback have focused on semantic rewards at the utterance level. |
| Approach: | They propose a multi-reward RLAIF framework for speech-in/speech-out dialogue systems . they combine semantic, audio-quality, and emotion-consistency rewards . |
| Outcome: | The proposed framework improves speech-in/speech-out dialogue system quality . it combines semantic, audio-quality, and emotion-consistency rewards . the proposed framework is available to download from the cdc. |
BatonVoice: An Operationalist Framework for Enhancing Controllable Speech Synthesis with Linguistic Intelligence from LLMs (2026.acl-long)
Copied to clipboard
Yue Wang, Ruotian Ma, Xingyu Chen, Zhengliang Shi, Morunliu Yang, Wanshun Chen, Huang Liu, Jiadi Yao, Xin He, Qu Yang, Qingxuan Jiang, Fanghua Ye, Juntao Li, Zhaopeng Tu, Xiaolong Li, Liefeng Bo, Min Zhang
| Challenge: | Existing approaches often fail to leverage the linguistic intelligence of Large Language Models (LLMs) Existing models lack the ability to follow text instructions for controllable Text-to-Speech (TTS). |
| Approach: | They propose a framework where an LLM acts as a conductor, understanding user instructions and generating a textual plan - explicit vocal features. |
| Outcome: | The proposed model outperforms open- and closed-source models in speech synthesis and achieves zero-shot cross-lingual generalization. |